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ABSTRACT 

The literature review vas undertaken to establish the 
current status of the aethodology for forecasting and assessing 
technology and for quantizing huaan resource parameters with respect 
to the iapact of incoaing technologies. The review of 1(fO selected 
docuaents applicable to the study was undertaken with eaphasis on the 
identification of aethods of assessment of the iapact on huaan 
resource parameters of advances in technology. The review failed to 
provide a solution to the problea of quantizing human resource 
parameters with respect to the impact of incoming technologies. 
However, the use of a normative forecasting technique was strongly 
supported by the literature. In particular, a relevance tree approach 
was the technique viewed as amenable to the problea of successive 
identification of increasingly finer components in an organized, 
structured manner. The relevance tree procedure known as a Design 
Option Decision Tree was identified as practical for detailing a 
system to a level permitting the identification and assessment of 
human resource components for impact quantification. A six-page 
bibliography and 17 expanded summaries of documents having major 
significance are appended. (Author/NH) 
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SUMMARY 



PROBLEM 

A research objective of the Air Force Human Resources Laboratory is the 
development of methods for defining the components of innovative technology 
and for measuring the effects of the incoming technology on Air Force human 
resources. The human resource parameters of concern include manpower (e.g., 
numbers, job types, skill levels), training considerations, and cost data. 
Earlier Investigations established the feasibility of using human resources 
data in design trade studies and of determining and graphically depicting the 
array of trade-off options iivailable before inception of hardware design. The 
purpose of the review reported here was to establish the current status of 
the methodology for forecasting and assessing technology and for quantizing 
human resource parameters with respect to the impact of incoming technologies. 

APPROACH 

A search and critical analysis of the literature was undertaken to review 
the status of forecasting and assessing technology and of techniques for pre- 
dicting the impact v technology on human resource parameters. The size of 
the literature base in the area of technological forecasting and technology 
assessment precluded the conduct of an exhaustive review. Rather, the deter- 
mination was made that inclusion of selected references which described 
current methodologies in detail and which were judged as representative would 
better satisfy the objectives of the research effort. Emphasis was directed 
toward location of quantification methods for human resources impact of 
technological innovations. 

Computer-based reference systems of the Defense Documentation Center, 
Ni?tion3l Technical Information Service, Systems Development Corporation and 
The Ohio State University were employed as integrants to a conventional 
search of the professional literature. 

RESULTS AND CONCLUSIONS 

A total of 140 documents applicable to this effort were reviewed. 
Seventeen :if these were identified as having major significance to the present 



effort. For these 17 » 3-pa8e expanded summaries were generated and appended 
to this teport of the results of the literature survey. 

Review of the literature failed to provide solution to the problem of 
quantizing human resource parameters with respect to Impact of incoming tech- 
nologies. However, the use of a normative forecasting technique was strongly 
supported by the literature. In particular, a relevance tree approach was 
the technique viewed as amenablo to the problem of successive identification 
of increasingly finer components in an organized , structured manner. The 
relevance tree procedure known as a Design Option Decision Tree was identified 
as practical for detailing a system to a level permitting the idencif ication 
and assessment of human resource components for impact quantification. 

DevelopiDent of quantizing methods and procedures for evaluating the 
adequacy of the evolved techniques in generating human resources data for 
system designers and personnel planners are highlighted as goal directions 
for the remaining study effort. 
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Section I 



INTRODUCTION 

The increasing demand to improve the capabilities of new weapon systems 
entering the inventory of the military is forcing a change in the traditional 
conceptualization of the man-machine interface and the support requirements 
of a system. Advanced manpower planning and restrictive budget constraints 
require the systems planner to refine the projected human resource estiroares 
to result in more optimal employment of the human resource component. It Is 
no longer sufficient to roughly approximate the human resource requirements 
necessary for operating and maintaining a given weapon system. Since a Riven 
weapon system usually Integrates technological innovations, it becomes essential 
to be able to predict the impact of these innovations on human resource param- 
eters. Military management must therefore develop a feasible way to locate 
and anticipate the advent of technological Innovations and predict, in advance, 
the human resource consequences. 

Historical evidence that such an action Is necessary when dealing with 
complex weapon systems can be found readily in prior Air Force procurements. 
A single anecdotal reference Is used for Illustrative purposes. In the intro- 
duction of the radar gunslght on the F-86 fighter in Korea, maintenance aspects 
of the subsystem were not given sufficient advance consideration. Consequently, 
the sight became operational in the field but could not be adequately main- 
tained. A priority effort was undertaken to train existing gunslght repairmen 
on the new system, a logical decision, since the additional training would be 
minimal for this group of personnel. Unfortunately, the training waa conducted 
at the manufacturer's plant, an action which effectively removed conventional 
gunslght repairmen from the field, further compounding an already serious 
maintenance problem.* 

In the present environment, a persuasively compelling argument for re- 
flecting the impact of technological innovation on human resources in the 
design phase is found In cost considerations. The largest single llfe-cvcle 
cost in a weapon system is the human resource cost over the period of time 



* Personal observation of the senior author. 
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iliM syKtrm is In the operational Inventory. A system designer must have 
access to dnla concerning the human resource parameters relative to various 
design options in order to select the most effective design based on life- 
cycle cost considerations. 

Eckstrand, Askren* and Snyder (1967) emphasized this need for a detailed 
consideration of human resources by classifying these resources as an "Impor* 
tant design parameter** but cautioned^ ^^unfortunately^ the data and technology 
for (this) involvement . . . have not been developed to the required degree/* 

In this paper the term technology will be used to describe the area of 
knowledge that composes the applied techniques used in the production of a 
weapon system. Therefore^ technology would encompass all disciplines Involved » 
e.g.» mechanicst electronics^ and fsetallurgy^ An innovation will be considered 
as any new aspect of a given discipline \^lch has potential practical applica- 
tion. What we are therefore seeking is a way to predict the Impact of a tech- 
nological innovation upon the human resource requirements of a specified weapon 
system. If one conceptualizes the process generally as was done In Figure It 
it can be seen that the design process Is viewed as making decisions based on 
the available technology to meet the objectives established. The objective 
may be singular or multiple so that the design process is restrained by exist- 
ing or emerging teclmology and the pre'-established design objectives. We 
are hypothesizing that the availability of a technological innovation would 
affect the design process » If the innovation is Implemented in the design, 
the result will be a different weapon system design. Each weapon system de- 
sign may have different human resource requirements^ 

The simplified diagram presented in Figure 1 Illustrates the basic 
considerations being addressed. Naturally » an innovation may be rejected as 
having no impact on design. Or the selection of the innovation may have such 
a small impact on the design process that no significant difference would be 
reflected in the weapon system produced. It would also be possible to have 
two weapon system designs which would have the same human resource require- 
ments. (Note: in thi» paper, the human resource includes not only the 
operator of the weapon system but also the support staff which maintains the 
total mai« "hlne system.) 
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The final goal would be the establishment of a technique which would 
allow a quantifiable description of an innovation which provides the capa- 
bility of predicting human resource requirements. Thisj action would effectively 
prelude the detailed design process and weapon system outcome stages and 
e&t«»blish the link between technological innovation and human resource parameters. 

The present research effort is a multiphased program directed toward 
the generation of methods for identifying snd measuring, prior to system 
development, the effects of technological innovation on the human resource 
parameters required to maintain and operate a weapon system which will in- 
corporate the technological innovations into its design. The ideal objective 
of this effort is to generate this data in forms useful to system designers and 
personnel planners to be considered in design decisions and manpower require- 
ment projections (see Appendix A-1) . 

Presented in the following pages are the results of the first phase of 
the effort, a search and critical analysis of the literature reviewing the 
current status of forecasting and assessing technology and in the techniques 
for predicting the impact of technology on human resource parameters. This 
survey is not an exhaustive review of the literature in the area of techno- 
logical forecasting and technology assessment. It was felt that the objectives 
of the study would be better served by the inclusion of selected references 
which discussed the current methodologies in detail and which were Judged to 
be representative of the field. For a more comprehensive grasp of this field, 
the reader is referred to Martino (1972), Ayres (1969), or Cetron (1969). In 
addition to the material contained in these references, a reader interested 
in a more extensive consideration of technological forecasting is referred to 
the bibliographies provided by Harrison (1971), 800 references, and to 
Chamberlain (1971), 108 references. 

A total of 140 documents applicable to this effort were reviewed. Of 
these documents, 17 were identified as being of major importance to the present 
effort. Expanded summaries were made of these key documents. These summaries 
arc includi'd as an appendix. 

Knipli.jsls of Lhis survt/y was dIrctU'd toward identification of methods 
for UHsessment of tht- impact on human resource parameters "f advances in 
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technology. Many of the usable and relevant documents described efforts per- 
f armed by or for the various branches of the military. Air Force sponsor eu 
studies predominated » a finding comparable to an earlier review of the litera- 
ture on prediction of personnel requirements (Folley, Fairman and Jones, 1960). 

In addition to a conventional search of the professional literature, the 
principal sources for document identification were the computer-based reference 
systems of the Defense Documentation Center 'DDC) , National Technical Informa- 
tion Service (NTIS) , the System Development Corporation INFORM and ERIC/ORBIT* 
and The Ohio State University computer-search systems. 
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Sect^on II 



FI'riDINGS 

METHODOLOGIES FOR PREDICTING AND ASSESSING TECHNOLOGY 

Sunuaarization of the diversity of methods for predicting a future state 
of technology appears best accomplished by classifying the many techniques 
under generic headings. The current body of Information available concerning 
technological forecasts and assessments reflects the merging of ideas from 
several formal disciplines. It is also apparent In reviewing the area that 
the objective of an assessment or forecast, which is always to predict the 
future, may vary in the sense of how the projected inforamtion is in itself 
used. The general concept of what is regarded as technology must also be 
considered. The transfer of scientific Imowledge into a technology or the 
adaption of an existing technology to a new use are two distinct processes 
which should be addressed. In the first instance the concern is with vertical 
transfer; in the second, horizontal transfer (Rooney, 1971). Whatever the 
concern, the Impact of technological change must be evaluated in the context 
of the outcome objective. The objective may be to establish the feasibility 
of some innovation or to resolve a decision problem. Due to the multi- 
disciplined development of this area an unwieldy conglomerate of methodologies 
has emerged which are used to project the future. As Lanford (1972) points 
out: 

The consideration of the subject of technological 
forecasting has been made more difficult by the 
differing terminologies that applied to the same or 
to similar methodologies, by the proliferation of 
'methodologies,' and by the discussions of 'intuitive,' 
'exploratory,' or 'normative' techniques. Further 
problems have appeared in considering techniques of 
resource allocation as forecasting methods. 

In attempting to address this problem, some taxonomic structure is required. 
Perhaps the most parsimonious attack would be to build upon the assumptions 
of a technological forecaLc. 

SinL-e direct knowledge of future events is unknown, there would appear to 
bo two primary assumptions underlying a forecasting methodology. First, data 
or inlormation about p£i.st events can be obtained and quantified, and second. 
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there Is some relationship between past and future events. The difference 
in methodological approaches therefore, largely hinges upon the type of re- 
lationship assumed between past and future events. 



A popular initial taxonomy Includes three major methodological approaches: 
intuitive, exploratory, and normative. The current state of the discipline in- 
dicates that these categories are not satisfactory. Several authors have 
pointed out that all approaches can be considered intuitive (Martino, 1972, 
Lanford, 1972a). The term intuitive therefore loses its function when con- 
sidered in the broader philosophical sense. The approach normally labeled ex- 
ploratory also has lost much of its value since so many approaches fall under 
this category. Therefore, a new taxonomy seems to be appropriate. It is sug- 
gested that there are basically two major approaches, that of projecting the 
future from available data (extrapolation), and that of establishing a set of 
desired goals in the future and working back to the present situation to 
determine the changes necessary to obtain the goals (normative). 

In addition to these major approaches, it is beneficial to consider the 
technique for gaining the basic data used in the projections. There appears to 
be fierce basic techniques employed in the extrapolation approach; judgmental, 
projection and analogy. In the judgmental technique, the expert opinion of 
individuals is collected in som»> fashion to establish a quantifiable value. 
In the projection technique, some relatively empirical measure is collected, 
and then a curve is selected to project future expected data points. In the 
analogy technique, some associated variable is linked to the variable of 
interest and its patterned relationship is used to determine the outcome. 

The normative approach appears to consist of two basic methodologies; 
network construction and matrix consideration. In the network construction 
approach, points of critical concern are identified projecting backwards from 
desired goals to the current situation. In matrix consideration, several 
vectors of interest are generated and a matrix of interactions is developed to 
establish a quantifiable value of each interaction outcome. Figure 2 displays 
the taxonomy with some specific methodologies displayed. [For a more compre- 
hensive presentation of methodologies, see Lanford (1972a), who has collected 
all the methodologies together for review.] 

Judgmental Forecast in/^ 

Although this paper considers the judgmental technique as an extrapolation 
ErIc "'"^^^^ consi|ged exploratory. The classification is made on 
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Figure 2. Technological Forecasting Taxonomy 




the bas?.s that these procedures are concerned with exploring the possible 
futures Inherent in present capabilities. Though all methods discussed up 
to this point are exploratory, the term "exploratory forecasting" in recent 
popular usage appears to be frequently associated with the Delphi procedure. 

Since its reported genesis In 1963 (Dalkey and Helner) , the Delphi 
procedure has been extensively used by those engaged in exploratory forecast- 
ing exercises. This popularity may be based on the fact that the Delphi ap- 
proach is designed to arrive at a prediction of what the future situation 
• will be, rather than a consideration of the many possible situations that 

could exist in the future. 

A second basis for appeal of the Delphi procedure may lie in its 
demonstrated reliability. Reliability here is used in the experimental sense; 
comparable results can be produced by repetition of the procedure with dif- 
ferent panels (Ayres, 1969). 

The Delphi procedure has characteristics which serN^e to differ- 
entiate it from other tools of the forecaster. It maintains anonymity 
of the participant experts throughout the complete cycle, thus obtaining the 
advantages possible from group interaction and at the same time obviating the 
disadvantages inherent in such an approach: the effects of vested interests, 
common biases, disproportionate influence of a dominant individual, and 
pressure to agree with the majority view (Gordon and Helraer, 1964) . 

The Delphi procedure, however, should not be viewed as the complete 
panacea to the forecasters* dilemma; for, as cautioned by Martino (1972), 
"Although the Delphi method ... Is both popular and practical, it suffers 
from the shortcoming that it is almost completely subjective in nature." 
A further criticism of the Delphi procedure is that it concentrates on one 
event at a time and thus does not allow a consideration of interactions 
atnong the events being forecasted. Reacting to this criticism, a procedure 
was devised by Helmer and Gordon, termed "Cross Impact Analysis," which can 
be used to arrive at an analysis of the interactions among events forecast 
by a Delphi panel or, for that matter, from any other source of forecast. 

ERIC 13 



Lanford (1972b) discusses the impacc study approach and relates it to 
the system analysis approach* Blackoan (1973) compares a cross impact model 
against a decision tree approach for long-range planning and asserts that 
the cross impact analysis has greater utility in that it reduces the com- 
plexity of the forecasting task. 

Projection Forecasts 

In this method of forecasting, the forecaster systematically records 
the past rate of technological innovation and advance in the area of interest , 
assumes this past: rate will continue into the fixture and fits an appropriate 
trend to the data, generally through the use of a regression technique. An 
example of application of this procedure can be found in Lamar (1969). This 
method of forecasting is subject to the same caveats that are expressed for 
statistical applications of regression analysis; for example eee Ostle (1963). 

A more complex approach is to select a curve fitting technique employed 
in other areas of research. This approach to forecasting has made use of 
systematic procedures developed by biologists to explain and predict the 
process of biological growth. The basis for this adoption lies in an observed 
parallelism between biological growth and "growth" in performance of techno-* 
logical devices. The biological growth curves have been used as conceptual 
models for technological development with the goal of obtaining greater ac* 
curacy of prediction. Lenz (1962) described this approach, but did not lend 
stronR support to it. Biological growth curves based on mathematical functions 
have also been investigated by forecasters. Two growth curves frequently used 
in technological forecasting, the Pearl-Reed curve and the Gompertz curve, are 
discussed by Martino (1972). >Iartino contends that neither method can be 
viewed as being completely rigorous. However, he does conclude that the use 
of growth curves is preferable to the Delphi method if sufficient historical 
data are present. 

Analogy Forecast 

Analogy forecasting is an attenpt to arri/e at a forecast by comparing 
the present situation with some similar technology in the past. A more 
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precise attempt In this area would be the establishment of a relationship with 
another innovation or variable which can be measured or quantified so that if 
the relationship is known» a projection caa be made based on the .in.iloROus in- 
dicator. An example would be the work of Mazlish (1965) who used the develop- 
ment pattern of railroads as an analogous indicator for the space program In 
exploring the process of projection by historical analogy. A number of problems 
are associated with this technique. One of the major problems is referred to 
as the error of "causal analogy" (0*Conner. 1970). This error is committed 
when one observes a few similarities between the past technology and the situ- 
ation he is seeking to forecast and, without gathering more evidence, concludes 
that the two situations are in fact analogous. More involved problems in fore- 
casting by analogy include the introduction of possible forecaster bias in his 
refusing to consider "implausible" outcomes. An associated problem is that of 
historical uniqueness. In this problem area, the forecaster is drawn into an 
erroneous conclusion, since his historical base is not representative of the 
past situation. These sources of error, as well as others, are discussed in 
more detail in Mazlish (1965). 

Network Construction 

The techniques of forecasting grouped under the descriptor "network 
construction" are, to a large extent, products of systems analysis procedures 
and emphasize the use of some quantitative approach. Cetron, Martino and 
Roepcke (1967) report on 30 quantitative methods applicable to research and 
development decisions. Normative methods attempt to identify and to locate, 
with reference to one and another, the elements of some system and to investi- 
gate relationships existing between the elements within the system. These 
techniques are amenable to, and are frequently used for, examination of system 
limitations, capabilities, costs, and design optimizations for identifying an 
approach leading to best attainment of a special goal. Three basic techniques 
of network construction forecasting which are the most typically used in fore- 
casting exercises are: relevance trees, mission-flow diagrams, and morphological 
models. 

Morphological models are used in the unique circumstance where no 
hierarchical relationships exist between the elements being considered and 
where each element can be altered without ensuing system ramifications 
(Zwicky, 1969). 
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•riM. u..;t' mlHsfoM-f low diaRrams is most practical for system descrip^ 
tion. The system is described in terms of branching paths where the 
description can be accomplished by use of one or more series of segmented 
steps or paths. This technique was oriRinated to permit the analysis of 
the elements of selected military missions. Probability of accomplishment 
or cost, as examples, can be associated with each step along the flow 
diagram, and the diagram can then be used to generate a normative forecast 
for another variable such as performance requirements to satisfy the mission, 

The relevance tree approach is typically used in a circumstance where 
specific levels of abstraction, or complexity, can be identified. Histori- 
cally, the relevance trt-e method was co.nmonly used for organized description 
purposes a.d had wide biological and botanical application. The initial 
application of a relevance tree approach to forecasting was apparently per- 
formed by Wells in 1958 (Cetron, 1969). 



A full-scale application of the relevance tree method was performed by 
Esch for the Honeywell Corporation. Continuing development by Esch and 
Jestice resulted in a procedure called "Planning Assistance Through Technical 
Evaluation of Relevance Numbers (PATTERN)" (Esch. 1965). In this approach 
"relevance" numbers are assigned to the options at each level of the tree. 
A path is selected through the tree and the product of the numbers on each 
branch of the path followed is obtained. This product is used as a figure 
of merit for comparison of this path against other possible paths in the 
tree. At each level In the tree, the relevance numbers assigned are pro- 
portions which satisfy a restriction that the total for the level roust sum 
to unity. This process is referred to as "normalization" of the relevance 
numbers and is a distinguishing feature of the PATTERN approach and, in 
ncnoral, ot the application of relevance numbers in relevance trees. A 
danger in following this technique lies in placing too much confidence in 
the significance of differences between the numbers obtained through such a 
procedure (Martino, 1972) (see Appendix A-2). As an example, it can be 
established quickly that the figure of merit for any specific approach in a 
simple tree having only two levels would be expressed to a hundredth. The 
decision with respect to the absolute size of a difference which will be 
accepted as significant therefore roust be approached with caution. 
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Overview 

Despite the plethora of proponents and techniques available for tech- 
nological forecasting, the majority view of those who have tried to assess 
this discipline seems to be something less than optimistic. Dory and Lord 
(1970) in an article asking the question. "Does Technological Forecasti««» 
Really Work?", concluded that, "Technological forecasting, the perception 
and interpretatlop of change, appears to represent a far more difficult 
problem than is currently indicated b> its proponents or addressed by their 
techniques." Another criticism was voiced by Bright (1970), "It ftechno- 
logical forecasting) remains inadequate for some of management's needs 
because: (a) ... ; (b) forecasts of technical capabilities do not necessarily 
deal with their diffusion, input requirements, and Impact." 

In a broader context, an Interagency Task Group on Technological Fore- 
casting in the Federal Government (1966) concluded, "There is little, if any, 
evidence of successful attempts to forecast actual 5 to 10 year requirements 
for skill in broad classes of occupations." In this same time frame, Hacke 
(1967) (see Appendix A- 3;, discussing the feasibility of anticipating 
economic and social consequences of technological innovation, talked about 
two aspects of forecasting, parameters and patterns. According to this 
author, the forecasting of parameters has been the subject of much work. As 
Hacke states, "The following examples demonstrate that there is no lack of 
methods for forecasting trends in socio-economic parameters:...." Hacke 
continues, "in the projection of economic and social patterns , by contrast, 
little has been done that goes beyond historical analogy and expert opinion." 
Earlier Schoeffler (1955) was more critical in his evaluation of explanations 
of changes in patterns by categorizing them as "story telling." Though this 
statement was made almost 20 years ago, the situation apparently has not 
been completely reversed. 

A summarization of the criticism of technological forecasting methods 
was succinctly made by Lenz (1969) in the following quotes. 

In practice it may often be observed that data for 
parameters of performance which would be useful for 
measurement and prediction of progress are very 
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difficult to obtain. Performance Is someti'nej? not 
measured at all, or Is often not measured at regular 
intervals . . ..Thus without a clear and specific 
knowledge of exact limits of present and past per- 
formance* it is not unusual to find a reluctance 
toward projecting or publishing forecasts of future 
performance in the same area. 

Lena continues, "The development of a useful quantitative forecast, using 
even the simplest methods of trend forecasting, is likely to become a 
lengthy research project." Lenz summarizes the products of technological 
forecasting by stating, "The scarcity of good examples (good forecasts) is 
compounded by the dlfflcluty of the search through large quantities of 
material labeled as forecasts of future trends to find the few which are 
useful." (Support for this statement may be found by consulting the bibli- 
ography compiled by Harrison (1971). The author identifies more than 800 
sources pertaining to tenlinologlcal forecasting spanning the years 1936 to 
1970.) 

Roberts (1969) (see Appendix A-4), critically appraising exploratory 
forecasting, concluded that most of the existing methods are variants of 
simple trend extrapolation proc«?dure8 and have limited utility In the current 
dynamic technological environment. 

This objection to the lack of devc>lopment of more than a simplistic 
approach method is also made by Jantsch (1967) In his statement that only a 
limited number of influencing factors are taken Into account and mathematical 
formulations "do not yet include even all of these recognized factors." 

Of the Delphi procedure, Roberts speculates about "whether a Delphic 
sampling of appropriate whiz kids would have predicted an effective inter- 
continental ballistic missile when Vannevar Bush failed in his prophecy!" 

Roberts argues strongly for the need of a "learning model" which would 
combine a real-time system, on which data could be collected and interpreted, 
with a forecasting model. This combination could allow "parameter and 
possibly even structural changes in the forecasting model based on experienced 
model ... performance." 

Normative forecasting is viewed by Roberts as failing to direct a suf- 
ficient amount of effort toward verification of the correctness of the 
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assumed relationships between allocated resources and generated technology. 
Supporting this statement is the Dory and Lord (1970) comment, "We agree with 
the literature that normative forecasting is a better answer than the explora- 
tory approach for many applications. We are not sure either attacks the real 
need for information." 

Roberts concludes his appraisal by calling for a technique integrating 
exploratory and normative forecasting methods and making use of feedback loops. 

In retrospect three points need to be considered. First, although there 
are a considerable number of techniques currently bei.jg used to forecast tech- 
nology, none of these are singularly more effective. Aa in any discipline 
they should be viewed as basic tools, each with Its own peculiar unique ad- 
vantages and disadvantages. To a degree the selection of a given technique 
Is predetermined by the purpose of the research. In addition, a consideration 
must be made as to the type of information that Is desired as a final outcome. 
Drawing upon Metroff and Turoff (1973), an adaptation of their conceptualiza- 
tion of information types is provided in Table 1. As can be seen, the type of 
consideration ranges from the relatively simple one of establishing the tech- 
nical feasibility of a specific short term goal such as the use of Wankcl 
engines In aircraft to the more complex long term goal exemplified by the re- 
duction of population growth over the next 90 years. The selection of a 
specific methodology may be strongly influenced by the type of outcome informa- 
tion expected. For example, type IV Informatioiv such as how to geographically 
allocate Air Force resources to maximize Air Force effectiveness In meeting 
national objectives, -jou id probably require some type oi matrix consideration 
In the attempted forecast. 

Second, the technique .-elected only provides a methodology not a certainty. 
The technique can only be asked to be effective if the data is in some way re- 
liable, valid and credible. The strongest technique cannot withstand the vib- 
ration of Inconsequential data. Therefore, a far greater effort must be placed 
upon data measurement and operational definitions of variables being considered. 
Prior to this time most technological forecasting and assessment has been based 
on either highly restrictive quantitative data; i.e.. weight versus thrust of 
engine performance, which may not reflect the conceptual variable of interest, 
performance (Alexander and Nelson. 1973), or on extremely general observations 
which lack the reliability desired in measuring a variable (Simmonds, 1973). 
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Table 1. Types of Information Obtained In 

Technological Forecasts and Assessments 

(Adapted From Mitroff & Turoff 1973) 



TYPE 



Feasible Technological 
Developments 



II Potential Applications 



III Significant Applications 



IV Potential Consequences 



V Policy or Resource 
Allocation Issues 



VI Potential Resolution 
of Issues 



DESCRIPTION 



Feasible usually means* in this context » 
technically feasible if the "required" 
resources are invested or available. 

This Is any possible application of the 
previous technological developments wlth-> 
out *'egard necessarily to values - i.e.» 
whether the application is good or bad. 

This is some subset of "all" ootential 
applications or a transformation to some 
set of significance to the intent of the 
study. 

Any consequences » good or bad, which may 
affect opinions of scenarios about the 
future » or our interpretation of the past. 

Concerned with the decision issues under 
examination or arising as a result of po- 
tential consequences of decisions. 

The controls that can be Imposed to effect 
the likelihood of various developments » 
appli cat ions » and consequences. 
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Third, what we are attempting tu forecast is in itself not generally de- 
fined. That is, technology is an extremely broadly interpreted word which can 
be anything from a broad scientific discovery, such as fusion, to a highly 
specific item, jet engines. In either case the research scientist is inter- 
ested in determining the impact of some perhaps almost unknown event in terms 
of a very concrete system or outcome. The very fact that someone may be con- 
cerned with the impact of the laser upon photography doss not in itself estab- 
lish any given approach. The technological innovation must be defined in some 
specific set of characteristics which in turn must be related to the photo- 
graphic characteristics of interest. The vast latitude between the specificity 
of forecasts creates one of the major hazards in any logical synthesis of avail- 
able research. This, perhaps, suggests that in order to study the question, a 
series of interrelated studies may be necessary to establish a conceptual net- 
work of data which will be the result of several different approaches to the 
problem. 

METHODS FOR RELATING STATES OF TECHNOLOGY TO THE IMPACT ON HUMAN RESOURCES 

Qualitative and Qua ntitative Personnel Requirements Information 

Through a process of evolution, the Air Force developed a concept for 
stating personnel and manning requirements of developing systems. This concept 
has undergone a series of modifications and name changes; it is currently known 
as Qualitative and Quantitative Personnel Requirements Information (QQPRI) 
(USAF, 1961). The QQPRI is a statement of projected manpower required to 
operate and tr maintain a new weapon system. It is usually based on a task 
analysis and is summarized in terms of an 'Wrage" operational unit. The 
summary lists the numbers and career areas of personnel required. It also 
provides an indication of the skill level of each individual. Folley, Fairraan 
and Jones (1960) conducted a survey of the literature on methods for prediction 
of Air Force personnel requirements. The overwhelming majority of the 121 
documents abstracted dealt with QQPRI and its antecedents. Only 1 of the 121 
reports surveyed addressed the question of elaboration of a procedure for 
estimating manpower requirements (Ray et a3 , 1957). This procedure used a 
rating scheme developed by Ghlselll and Brown (1955) which classified skills 
into 3 levels of complexity. The approach was based on use of the same inputs 
involved in QQPRI generation and appeared to result in the same product, speci- 
fication of QQPRI data. Thus, the Ray procedure presented no basic Improvomont 



-^ver the QQPRI method* The reader interested in more detail concerning QQPRI 
is referred to Losee et al (1961) and a later effort, Dpmaree et al (1962), 
which detailed the approach methodologies used to produce QQPRI and made dis- 
tinctions between QQPRI and other products in the Personnel Subsystem effort* 
Both documents w^ere intended to serve as guides to the preparation of QQPRI for 
use in weapon system development efforts. 

A look at extra Air Force attempts to establish procedures for specify- 
ing personnel and training requirements for incoming weapons systems is pro- 
vided by Rupe (1963). Haines and Gael (1963) (see Appendix A-5) conducted a 
survey of manning estimation techniques used in defense industries. In all 
cases surveyed » QQPRI was used as the approach method. The resultant QQPRI 
products were found to be based on five methods of obtaining input data. 
These were: expert estimate, historical comparison, task analysis, **sovereign 
factors'* (e.g., cost used as an index of complexity together with a queuing 
theory approach), and modeling approaches. Essentially, the QQPRI concept 
has not changed since these early reviews. 

The search for a method to specify system manning requirements earlier 
in the development cycle of the system has led to investigation and experi- 
mental application of simulation techniques with AFM 66-1 (1972) type mainte- 
nance data along with ocher inputs as the information b^se. These procedures 
to date are not documented in published material. t 

Simulation Techniques 

The feasibility of specifying manning requirements of advanced systems 
by means of matheraatlcaX laodeling approaches has been Investigated for at 
least a decade. Barton et al (1964) (see Appendix A-6) used two mathemati- 
cal techniques (queuing and a linear programming model) to compute manning 
requirements and training needs in a developing system. The authors pointed 
out that, "in practice, failed systems or units pile up in lines waiting for 
service or men are incompletely utilized." They felt this model adequately 
dealt with these conditions. A set of queuing tables for use with this 
model was generated by Purvis, McLaughlin and Mallory (1964). Further 

t Personal communication with R.J. Schlffler, (USAF.ASD). The interested 
reader is referred to AFHRL prepublication draft: York, Michael L. (Major, 
USAF), A-10 Manpower Requirements Development Test, 14 September 1973. 
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investigation of this application of queuing techniques was made by Purvis, 
Mallory and McLaughlin (1965) (see Appendix A-7) with the intent of estab-- 
lishing the validity and reliability of the procedure for predicting manning 
requirements for weapons systems. The approach used by the authors was the 
application of the procedure to an operational system (the F-105D fire con- 
trol system) and to a system in development (the C--141). The conclusion 
reached was that the approach was usable and that '*the technique clearly 
establishes the validity of the finite cyclical queuing process as an accu-* 
rate description of the F-105 fire control system 

Snyder and Askren (1968) (see Appendix A-8) support the statements of 
Pur vis » Mallory and McLaughlin with their conclusion, '^queuing techniques 
are valid and reliable for predicting requirements and making trade-offs 
bfcitveen manning and personnel skills . * . 

Approaches taken by other investigators to simulation model solutions 
f:or specifying the impact of technological innovations on human resource 
parameters have included Bradley (1968) and Floyd (1968). Bradley suggested 
a Monte Carlo simulation model emphasizing prediction of manpower and spares 
requirements. This model was computer based, but provisions were made for 
manual application. A different modeling approach was advanced by Floyd » who 
suggested a mathematical model leading to the development of trend curves 
for prediction when two conditions were satisfied; firsts an ultimate limit 
could be identified and, second* when at least two data points were available. 
Current efforts under Air Force direction include the experimental effort 
mentioned in the section above » which is an approach entailing mathematical 
modeling of future aircraft systems* human resources requirements based on 
AFM 66*'l maintenance data assembled on a current comparable-mission aircraft. 
This effort resulted in the development of a computerized model which con- 
siders operations sc: ^arios and task data to project resource levels of tnen, 
spares and aerospace ground equipment (AGE). The research has bean completed 
and a series of reports are being drafted. The technique has been adapted 
for some current Air Force syst^is. 

Statistical Approaches 

Closely related to modeling techniques are statistical approaches to the 
prediction of the impact of technological innovation on human resources. The 
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statistical approaches to this problem have, to a large extent, emphasized 
regression analysis procedures to arrive at an estimate of the effects of 
equipment design characteristics on skill requirements or job performance. 

Meister. Finley and Thompson (1971) used a multiple regression analysis 
as one of the steps in a process which they termed "a sort of ^poor man's 
factor analysis'." For a summary of the procedure advanced by these authors, 
refer to the expanded summary included in Appendix A-9. 

The authors identified four factors as prime determinants of maintenance 
performance. These were accessibility, amount of diagnostic information, 
equipment configuration, and technician capability. 

The expected recommendation for replication with an expanded sample size 
to increase confidence in results was made. An interesting conclusion enumer- 
ated, but not emphasized, by the authors was the suggestion that length of 
experience after skill level qualification was not an important design param- 
eter. In view of the potential training and design consideration impacts 
represented by such a condition, any replication of this study should direct 
specific attention to collection of data addressing this issue. 

Smith, Blanchard and Westland (1971), investigated subjective techniques 
for generating human performance repair time data required for predictions of 
maintainability of proposed equipment. Data generated was compared against 
AFM 66-1 maintenance data by means of regression analysis techniques. The 
relationship was found to be linear (y = 1.2756x + 0.7021) with a positive 
correlation of 0.86. The authors concluded that subjective judgment could be 
used to predict maintainability of future systems. 

The relationship between equipment design characteristics, training costs 
and difficulty, and job performance was investigated by Lintz, Loy, Hopper 
and Potempa (1973) (see Appendix A-10) . The approach was through the use of 
a principal components (stepwise regression) analysis to derive prediction 
equations for performance time, errors, training time, and training equipment 
cost. Variables correlating significantly with performance were subjected to 
a factor analysis. This procedure resulted in the identification of" six 
factors. The authors concluded that the procedures followed provided a 
quantitative method for predicting the variables under study and that such 
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predictions could be used as considerations in a system design process. The 
authors call for validation of the procedure through following a system design 
effort from conceptualization to operational use. 

A subsequent study employing the same methodology was conducted by Lintz* 
Loy, Brock* and Potempa (1973) (see Appendix A-11). Organizational and 
intermediate levei maintenance data were collected on 10 avionics subsystems. 
Within these subsystems, selected equlpmeiit design characteristics were either 
measured or subjectively rated. In addition, 16 personnel variables were 
used. Maintenance tasks of differing levels of difficulty were used for 
estimating performance times and error probabilities for both high and low 
skill level personnel. 

Design, personnel, performance, and difficulty variables were intercor- 
related and were entered into regression and factor analyses. Equations 
were derived for predicting maintenance task difficulty and maintenance per- 
formance times. Maintenance performance time was predicted through a con- 
sideration of personnel and task difficulty variables. 

Multiple correlation coefficients (Rs) for predicting times from design 
characteristics ranged from 0.87 to 0.93. Task difficulty coefficients 
ranged from 0.73 to 0.80. Factor analyses on performance, personnel and 
design variables yielded some 17 factors over the different maintenance 
levels studied. The authors concluded that the results supported the merit 
of such an approach to predicting maintenance performance times and task 
difficulty and felt an extension of the procedure to additional avionics 
equipment was in order. 

A factor analysis approach was also used earlier by Topmiller (1964) as 
an approach method for conducting human engineering analyses and predicting 
system maintainability. A more global approach was employed in that the 
author was primarily concerned with equipment classes rather than directing 
attention to the LRU level - the object of study in Lintz, Loy, Brock and 
Potempa (1973). Topmiller reported that increased predictive efficiency was 
experienced with his approach when equipment was classified into homogeneous 
sets. , 
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Cross Projection Methods 

Most frequently encountered in reviewing the literature are studies of 
the effects of technological change on skill level where skill level is 
treated only in terms of gross projections* 

Crossraan (1960) provided a scheme for study of 'jypes of automation which 
involved classifying the types into continuous flow production^ prograaaaed 
machine^ and centralized remote control activities. One of the impacts of 
automation, according to Grossman, would be the developing requirement for a 
multiskilled maintenance roan, a skill type he alluded to as a "polyvalent 
cr^^ftsman." Davis (1962) concluded that levels required for semiskilled mass 
production jobs would be suitable for operators of automated equipment « 
Beaumont and Helfgott (1964) reporting on a study of 36 K!ompanies concluded 
thut worker skill requirements were fundamentally unchanged by technological 
innovation* Bright (1966) arrived at the disparate conclusion that automated 
machiner: tended to require less operator skill* Grossman and Laner (1969) 
(see Appendix A-12) separated skills into direct and indirect labor and re- 
ported finding an increase in direct labor skill requirements but concluded 
that with the inclusion of indirect labor into the consideration there was no 
overall increase in mean skill levels required as a result of technological 
change. 

Kraft (1970) called for study of specific skill responses required at 
various stages of automation. He felt that such an approach miight provide the 
information needed for adequate manpower planning. 



Relevance Tree Approaches 

The earliet referenced efforts of Wells (1958) apparently represented 
the first application of a relevance tree approach to the problem of the tech- 
nological forecasting for impact considerations. Continued attention was 
given to this procedure by Esch and Jestlce (Esch, 1965) with the result that 
"relevance nun>bers** were attached to the branches of the tree to permit se- 
lection of an optimized path through the tree. 

Hwang (1970) was one of the early investigators to perceive the relevance 
tree approach as a vehicle for study of specific weapon system prediction 
€|uc»Ht Ions. Hwanj* atU'mprcd to OHLablish tho comparability of risk analynis 
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to a systems analysis approach. He used a decision tree approach in consort 
with methods of risk analysis to predict risk in materials acquisition for 
the U.S. Army. 

In contradiction to a conclusion reached by Meister and Farr (1967), 
Lint^, Askren, and Lott (1971) reported that design engineers did consider 
human resources data if it was available and that when considered, this data 
did affect resulting designs. The authors recommended that human resources 
data be included as one of the design parameters in weapon system development. 
In seeking an answer to the problem of quantitative specification of human 
resources impact data in advance of the actual application of a technology, 
one of the above authors, Askren, in collaboration with Korkan (1971) (see 
Appendix A-13) investigated the feasibility of the use of a relevance tree 
approach as a vehicle for relating human resources data to design decision 
alternatives. The relevance tree developed by the authors reflected weapon 
system decision options and was termed a "Design Option Decision Tree (DODT)." 

The underlying basis for the development of this particular research 
approach apparently lies in the fact that in the design of a weapon system, 
the system designer can rely on a store of past engineering data which can be 
considered in selecting the hardware design option to follow. However, as 
these authors succinctly indicate. 



In the case of human resources data, there is no such 
bank of historical data. Thus, the need exists for 
the generation of data descriptive of Air Force human 
resources implications of the various design options 
identified in the decision trees. The generation of 
these data likely will require a combination of efforts, 
such as collection of historical and field data, and the 
generation of data through simulation and psyche physical 
procedures. 

The authors concluded that design option decision trees appeared to provide a 
vehicle for permitting a systematic consideration of human resources factors 
Involved in a specific design decision. 

In a follow-on study, Askren, Korkan and Watts (1973) (see Appendix A-IA) 
investigated the feasibility of measuring the sensitivity of human resource 
parameters data to design trade-off problems reflected in a DODT. Eight 
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tradonoff problems were chosen from an aircraft propulsion tree and psycho- 
metric data collected concerning the effect of these design options on 
selected human resources topics. The authors concluded that the study demon- 
strated the feasibility cf the use of psychometric methods in conjunction with 
a DODT to develop selected human resources data for use in system design 
decisions. 

Other Approaches 

The attempts to relate the effects of technological innovation on human 
resource variables over the recent past appear to have emphasized rather re- 
stricted avenues of investigation. The approaches represented by QQPRI gen- 
eration, regression analysis, queuing and relevance tree techniques dominate 
the relatively sparce literature in this area. Isolated Investigations of 
the feasibility of application of other approaches are found, but generally 
are not the subject of more than one investigation. Thus, in 1961 » Thomas 
et al (see Appendix A-lf) described a method they had developed for generating 
data on the effects of various levels of automation on human performance In a 
man-machine system. The method was advanced as a means of studying system 
control problems anticipated in the design of an adv^anced system. 

Grossman tl965a) reported on an approach intended to define skill level 
on a scale independent of the content of the skill. The approach suggested en- 
tailed measuring the man-hours expended in developing the skill resulting in a 
scale which would be "divisible into some six or seven levels." The Grossman 
scale basically would represent a parallel system to the Air Force scheme of 
skill levels and would be developed on a less satisfactory data base. 

In 1969 (Meister et al) , an investigation was conducted concerning the 
design engineer's concept of the relationship between design characteristics 
and technician skill level. The authors asked their sample of design engineers 
to rank ten statements of "different levels of maintenance skill" along a 
ten-point scale. They concluded, "Engineers claim to be capable of differ- 
entiating ten levels of personnel skill." 

Dean (1959) in an unpublished dissertation, reported on an investigation 
of the "primary Intellectual, perceptual, visual, interpersonal, procedural 
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and muscular skill requirements" imposed by the introduction of automated 
equipment in seven case studies of industry. A Job Description Check List, 
developed by Storey (1958), was used to provide quantitative indices of the 
skill components listed above for each case studied, both before and after 
automating. Dean found no effects of automation in 75 per cent of the obser- 
vations. . In the remainder of the observations, the number of skill Increases 
were approximately equal to the number of decreases. 

Schulze (1971) reported on an attempt to relate equipment design features 
to selected psychophysical indices. Though finding no statistically signifi- 
cant differences, the author reported strong trends for a majority of the In- 
dices to be higher during the mora difficult tasks. 

From the information available it appears that the relationship between 
existing states of weapon systems and human resource parameters has only been 
addressed in a crude and unsystematic manner. The prognostic needs of the 
armed services forged the concept of QQPRI. but this concept has lacked a 
specialized technique which would provide adequate information. Some of the 
methods used establish acceptable projections of manpower but are usually 
applied after the weapon system has transitioned into the production phase. 
Initial efforts have shown that human resource parameters will affect weapon 
system design if the impact of a given trade-off has reliable data provided. 

There is presently no body of information that would provide the required 
relationships between design characteristics and human resource parameters. 
The design option decision tree technique provides a network Instruction tech- 
nique that allows us to- reduce a system problem into a series of functional 
decision points. This certainly provides insight into complex system prob- 
lems, but it does not establish the necessary relationships. It is concluded 
that the methodology currently available in the area of predicting human re- 
source requirements for a given system is not any more advanced than fore- 
casting methodology, and it deals almost exclusively with the three manpower 
considerations of quantity, job type and skill level. 



Overview 

Research attempting to relate the Impact of technological innovations on 
human resources has, to a large extent, emphasized the investigation of the 
effect of automation on overall manpower skill requirements. The more com- 
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plcx issue of anticipating the specific changes which application of a tech- 
nological advance generates within a particular skill classification has been 
neither adequately treated nor resolved » 

An extensive review of earlier efforts directed toward assessment of the 
impact of automation on manpower skill requirements was conducted by the 
International Labour Office (1965). This effort reviewed 160 studies, 49 of 
which were accomplished in the United States. Each of the studies was cate** 
gorized in terms of the kinds of technological change involved and was class- 
ified with respect to its effect on selected social^ economic^ occupational, 
and personal factors. The studies in this review, and many of the more re*- 
cent studies, have dealt with skill requirements in terms of gross census** 
type predictions. 

In criticism of earlier work in this area Grossman (1965b) concluded, 
**To this date, we have essentially no valid models which will predict the 
rate of progress of automation in any single case, nor the likely changes in 
skill input." 

That attention is still directed at the level of gross projection is re- 
flected in sources such as Buechner (1972) (see Appendix A*16), who con- 
cluded that technological change and associated automation served to reduce 
demands on labor, but cautioned that the nature of his data did not allow 
consideration of individual jobs. 

DeGreene (1974?) asserts that in spite of 20 years study, a categorical 
statement concerning the effect of automation on skill requirements cannot 
be made* 

A major criticism leveled against QQPRI has been the relatively late 
stage in the system development cycle at which personnel information became 
available. Attempts to develop specific human resource information at earlier 
points in the system development cycle have led to investigations pursuing 
the approaches summarized in the sections above. To date, no approach has 
provided the answers sought. At least one individual (DeGreene, 1974?) ad- 
dressing this area of concern, pessimistically concludes, "... it will prob- 
ably be at least years before a predictive or even an explanatory theory of 
technological changes (as related to manpower resources) is developed 
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DeGreene calls for development of a systems theory integrating the various 
approaches of psychology, engineering, computer sciences » economics, and 
other related fields to deal with this problem. 

POTENTIAL METHODS FOR PREDICTION OF IMPACT OF TECHNOLOGICAL INNOVATIONS ON 
HUMAN RESOURCE PARAMETERS 

Esch (1965) reported on a modification of the PATTERN relevance tree 
undertaken by the Honeywell Corporation in an attempt to Identify present and 
future needs in bio-medicine. This effort was referred to as "Project Medi- 
cine." The term "medicine" in the title was an acronym standing for medical 
instrumentation and control identified and numerically evaluated. This pro- 
cedure used the decision structuring and relevance number assignment tech- 
niques developed for PATTERN but redefined the levels within the tree. For 
example, in the hardware trees, the primary activity level is the task level. 
In the bio-medical tree» this level was concerned with such activities as diag- 
nosis, palliative procedures, curative procedures, and prevention. System 
and subsystem levels of the hardware oriented tree became major body activ- 
ities (e.g., blood circulation) and body organs, respectively. In this 
fashion, the relevance tree was employed by Honeywell to plan a research and 
development program in medical electronics, thus providing a demonstration of 
the generalizability of the relevance tree approach. 

Investigating the relationship between advanced technology and work group 
behavior in a replication study, Taylor (1970) reported support for earlier 
conclusions that a pattern of greater peer leadership was more evident in 
groups exposed to more advanced technology and that an advanced technology 
climate facilitated planned social change efforts. 

• Bezdek and Getzel (1973), using gross projection methods, predicted the 

skill requirements for three future socio-economic environments: a status 
quo, a social-welfare, and a defense economy. They concluded that for all 
three futures manpower requirements at upper skill levels would be sub- 
stantially increased. The choice of national priorities was viewed by the 
authors as exercising a significant effect on skill types within job famil- 
ies. This factor was viewed as a compounding problem making accurate pre- 
diction of education, vocational and skill requirements very difficult. 
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The application of th.* material in this and in many of the other artl- 
clc« in this area is restricted by the same problem encountered earlier; 
most articles address themselves to trend extrapolations and gross predic"^ 
tions {o^g.f Gordon and Amenta 1969) » thus provide little useful insight for 
answer to questions concerning specific impact effects on human resources. 
This sentiment is supported by Hacke (1967) who stated » **lt is not feasible* 
however^ to forecast quantitatively the trends in a (social or economic) par- 
ameter or pattern with the certainty that the outcome will fall within the 
stated confidence limits 

QUANTIFICATION OF HUMAN PERFORMANCE FACTORS IN SYSTEM DEVELOPMENT 

Isolated investigations of the problem of quantification of human per- 
formiince in developing weapon systems surfaced as a result of the review of 
the literature. Certain of these may have application to the present effort 
and are identified for consideration. 

A detailed examination of thit' topic was undertaken by a 1964 symposium 
on quantification of human performance* The December 1964 Buman Factors 
Journal » a special issue » was devoted to selected (and edited) papers pre- 
sented at the symposium and workshop held at Albuquerque, New Mexico » In 
August 1964. This issue contains a theoretical article by Leuba (1964) 
dealing with human parameter quantification methods* In this paper » Leuba 
provided examples of the types of quantifications required in roan-^machine 
systems* Leuba associated selected equipment design procedures (e.g. task 
analyses^ trade-off studies) with the different types of measurement scales. 
As example; he stated » "Trade-offs imply the existence of ratio scales* 
This fact accounts for the current difficulty engineering psychology is ex- 
periencing in performing reasonable trade-off operations with systems 
engineers" ♦ Leuba then progressed to a summary discussion of methods of 
quantification which could be applied to man-machine system problems. 

Unfortunately > neither Leuba^ nor any of the other symposium participants 
could suggest a procedural solution to quantification of human performance 
factors in system development. Ten yeari. later, a suitable methodology for 
providing answers is still needed. As DeGreene (1974?) states "To date there 
has been no cohesive examination of the interrelationship between new 
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technology and manpower resources. ...abilities and skills must be better 
related to tasks* Jobs* and occupations. This is the ongoing challenge to 
human factors." 

In this last decade, a number of approaches to the problem of quant if y-> 
ing human performance factors have been attempted* Rook (1964) reported on a 
method for evaluating system failure performance based on rank-order data. 

Irvin, Levlts and Freed (1964) reported on the application of a sub- 
jective method of obtaining human reliability data. The approach Involved 
the use of a Llkert-type scale to obtain reliability estimates for 60 
maintenance tasks. In this approach, the authors related their findings to 
performance measures for operation of electronic equipment which were developed 
by American Institutes for Research (Payne et al, 1962). 

Kaplan (1967) reported on the results of investigating several types of 
scales for obtaining value judgments from subject matter experts. Ratio, 
difference » direct » and paired comparison techniques were selected for study. 
The author selected the difference scale based on the fact that it had a 
standard t^ich served to stabilize the scale in the absence of a known di- 
mension for the measure being considered* 

Finley et al (1970) performed a technical review of human performance 
prediction in man-machine systems* As a part of this effort » *'over 500'* 
performance prediction tests were reviewed. The authors felt that develop- 
ment of a usable human performance prediction technique dictated adherence to 
certain essential theoretical and methodolical requirements which they 
identified as: 

1* Accomplishment of a more exact understanding and description of 
system and behavior ial phenomena. 

2. Recognition that the prediction problems are multi-dimensional and 
multi-^level • 

3. Utilization of modern test development techniques (e.g. utility 
analysis.) 

The final conclusion reached by the authors echoed the general concensus 
found in earlier research^ "Human performance prediction tests can serve a 
great potential future role in the understanding, prediction ♦ and control of 
human performance in man-machine systems; but only to the degree that many 
current theoretical and methodological problems are resolved/' 
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SUMMARY 

This review of literature addressed the topics of technological 
forecasting methods and techniques for assessing the effect of technological 
advances on human resources. The guiding philosophy was to identify techniques 
and measures wliich could be applied in seeking resolution to the problem of 
specification, in advance of weapon system development, of the effects of 
technological innovations on the human resources required to maintain and 
operate the weapon system, and of producing this data in forms useful to system 
designers and to personnel planners. 

In order to provide this substantive analysis, a set of evaluative 
criteria was generated against which the identified studies were judged. 
These criteria were: 

• 

1. Does the method have application to the present re9earch problem? 

2. Is the method applicable in totals or are only selected elements 
appropriate? 

3. Does the method produce results couched in quantitative terms? 

4. What limitations are imposed by the method when considering such 
variables as 

— cost 

— time 

— ease of application 

— computer feasibility? 

5. Are there any evaluative criteria associated with the proposed 
method; for ex.'^inple^ can statements be made concerning 

— probability of success with method 

— levels (and limits) of confidence 

— validity of the procedure? 

The review of the literature failed to reveal a procedure which could be 
applied in toto to produce the answers sought under this research project. 
However^ the review of the methods and techniques subsumed under the topic 
••TiThnt^loKii' Kt»rt^\isl luji;" did sn^Kost possible approaches having merit for 
Uu* pri^siMii task* 

Tfchno logical forecasting is a young and vibrantly active field 
attracting participants from many professional disciplines. The sheer 
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number of practitioners and their resultant documentation compound the task 
of objective evaluation of the merits of the proffered approaches. However* 
among those attempting to appraise technological forecasting, a conse.isus 
was apparent favoring a normative approach to forecasting. This approach 
was felt to permit the reduction of an objective to a series of smaller 
choice considerations which were more responsive to some form of quantifi- 
cation. 

Review of the available normative techniques to technological forecast- 
ing led to the conclusion that a technique amenable for study of the problem 
under investigation is a modification of the relevance tree method known as 
a Design Option Decision Tree. The relevance tree approach has been demon- 
strated to be a useful vehicle for carrying out successive identification of 
increasingly finer components. A basic strength of this method lies in the 
fact that it forces an organization and structuring of the problem and helps 
to assure a complete consideration of alternative methods for solution of 
the topic under study. With the design option decision tree, any number of 
solutions to the overall problem can be traced and evaluated in terms of 
their requirements on the resources and technologies required for implemen- 
tation. Lastly, the use of this method provides an easy means of determin- 
ing If a specific solution has been overlooked. 

The design option decision tree procedure must have some method associ- 
ated with it for quantitatively assessing the impact of the technologies re- 
flected at various points in the tree on the human resource parameters which 
would be involved with each decision option. 

Since no solution to the requireaent, quantizing the impact of techno- 
ogical innovations on human resource parameters, was revealed as a result 
of the literature search, it is felt that while the DODT will provide con- 
siderable insight into the system problem^ some other methodology must be 
used in association with it to provide the data required. The few studies 
located ancillary to this topic investigated selected psychometric tech- 
niques but offered no procedural solutions. The selection, or development, 
of a method to quantize human resource impact predictions roust be an empha- 
sized goal of the present study. 

Attempts to provide assurance of the utility and validity of any tech- 
nique developed in this study must be included as a part of the overall study . 
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As suggested by Roberts (1969) . and in part employed in an earlier study 
(Purvis* Mallory and McLaughlin, 1965) » an inherently attractive approach to 
estimate the validity of any proposed technique would appear to be the com- 
bination of an experential model and a forecasting model into an overall 
procedure teni,ed a learning model by Roberts but perhaps more accurately re- 
ferred to as an "Adaptive Model." In brief, this model would combine a real- 
time system, on which human resources data would be collected and interpreted, 
with a forecasting model. Human resources data for the real-time system also 
would be generated by the forecasting model. Comparison of the two data sets 
would permit modification of the forecasting model until a parallelism of data 
was achieved. At this point the forecasting model could be employed with 
specific areas of technological advance. 

Another approach synthesized from the literature review might be 
classified as an "Incremental Approximation" method. This approach is sug- 
gested by the procedure followed in studies reported by Grossman and cohorts 
(1966 and 1969) (see Appendix A-12 and A-17). The procedure basically would 
represent a refinement to the adaptive model proposed in the last paragraph. 

This approach is conceptualized as entailing the selection of a tech- 
nology area which has experienced recent application of some technological 
innovation. The area selected roust have available instances of both the 

« 

Static and the innovative technology in operation. Empirical data Is collected 
on the operation employing static technology • The nonaative predictive pro- 
cedure and quantification method is independently applied to this same tech- 
nology and refined as described under the adaptive model* The revised 
prediction procedure is then applied against the operation incorporating the 
innovative technology and the results validated against empirical data col- 
lected from field operational experience with the innovative technology • 
After making any indicated refinements » the prediction procedure is then 
ready for application to conceptual systems employing anticipated technology. 

Reiterated, the overall research goal of this study is the development 
of a procedure whereby a family of existing or newly generated techniques 
and measures can be applied to determine in advance of actual system develop- 
mentt the impact of new technology on the human resources required for 
operation and maintenance of the system* 
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The next task in the conceptualized study approach is envisioned as the most 
difficult undertaking of the entire effort. This view finds support from a 
number of sources (Leuba, 1964; Grossman, i963b; Finley, et al, 1970; and 
DeGreene. 1974?). Specifically, this next task is that of developing 
measures and techniques permitting satisfactory levels of quantification of 
the impact of specific technological innovations on selected human resource 
parameters. The avenues of approach requiring pursuit are: 

• modification of an existing fechnique for human performance prediction 
to make it amenable to the present requirement . 

• integration of two or more existing techniques to result in a syn- 
thesis having applicability to the prediction task, 
development of totally new techniques or measures. 

The level of success achieved in this task will influence the remain- 
ing study objectives: The application of the resulting measures using the 
approaches suggested earlier in this report, and an evaluation of the value 
and utility of the procedure to system designers and planners. 

The absence of objective criteria for assessing the procedure dictates 
a reliance on expert judgment. The use of such a subjective source of eval- 
uation requires great care in the shaping and conduct of the information 
gathering process. This fact is clearly recognized and as the study pro- 
gresses, attention will be directed to any plausible way of reducing the 
subjectivity of the process. 

The authors are intimately aware of the exploratory and feasibility na- 
ture of the present undertaking. However, the importance of this approach 
should not be underestimated. As indicated by Darracott et al (1967), "... 
the current technology race places a high premium on the ability to assess 
developing technological trends correctly. The pace of this race and the 
increasing complexity of the problem have exceeded the ability of military 
forecasters to assess these trends by intuitive methods alone/' In the 
remaining stages of this study, selected avenues of approach will be pursued 
in an attempt to develop an acceptable method permitting a system designer 
to conceptualize the probable impact of and manipulate values for selected 
human resource components in designing cost-effective advanced military 
systems. Should no method proposed in this study result in a procedural 
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solution to this decades old problem, it is believed that a contribution 
nevertheless will have been made by this endeavor in that other researchers 
will be able to stand on our shoulders in their continuing search for a 
solution. 
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A-1 

Minister, D. , The? tmgact of Manpower Requtreroents and Personnel 

Sullivan* D. J.» R esources Data on System Design 

Askren, W. B. 



The importance of human factors to system performance has been shown a 
number of tirot^s. Both logically and empirically, the negative effect of in- 
adequatt* consideration of the human element during design can be demonstrated. 
The contribution of human error to the unreliability of overall system per- 
formance has been graphically and empirically illustrated. Evidence is 
accumulating that the cost of the personnel to operate and maintain a system 
throughout its useful life is equal to or exceeds that of the hardware. Thus, 
we have a double-edged problem, performance decrement and high costs, which 
can be related to the human element of the system. This problem has been 
made more severe by a history of development of new systems with emphasis 
primarily on the design of hardware and with little or no regard for the 
capability t>r cost of the personnel that will be available to support the 
Hyslera. U h.is bet-n aJvoc.itcd that one way to reduce the problem is to de- 
velop systiMiis with hiinuin factors data included as design requirements. This 
study is» therefore, directed toward a controlled examination of the utility 
of PKD (Personnel Resources Data) in system design and to the determination 
of the conditions under which PRD can be effective in influencing that design. 

GENERAL RESEARCH APPROACH 

The major purpose of this study was to determine the effect on system 
design of using manpower and personnel resources data as design requirements. 
Secondary objectives were to determine under what conditions and in what 
form these data should be useil to have maximum effect on design. Equipment, 
manpower data (e.g., quantities and skill levels), and PRD inputs (e.g., task 
information) which were produced during the development of the Titan III 
propel lant transfer and pressurlzation subsystem were adopted and presented 
incrementally to six design engineers to simulate the Air Force phase lA/lB 
development of that subsystem. Subjects were required to create schematics, 
equipment descriptions and drawings, control panel layouts, operating pro- 
tcdurt's .mil hills of material. Cost-effectiveness measures including 
.'ijiiipnitnt cost, equipment reliability, human reliability, system safety and 
tie.-; inn .ulecju.ie v were applied to the data. 



AN.A LYSIS 

The research strategy developed by Meister and Farr (1966) and Melster 
and Sullivan (1967) involves placing the engineer in a realistic design situa- 
tion in which ho must solve a series of design problems by using informational 
inputs related to these problems. In adapting this general methodology to 
the present study, the ^-""-wing steps were performed: 

1. Selection of an already operational subsystem which could serve as 
a model subsystem for the development of test inputs. 

2. Selection of appropriate engineer-subjects skilled in design of tlie 
type ui subsystem se lee ted. 

S3 
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3. Determination of the equipment and PRD Inputs which are character- 
istically provided during the system definition phase of development. 

4. Development of manpower and personnel resources data inputs. 

5. Determination of the sequence in which these inputs should be 
provided. 



6. Determination of the design responses and outputs which the engineer- 
subjects should supply in attempting to solve the design problems. 

7. Determination of specific measures which could be used to answer 
the questions which initiated the study. 

FINDINGS 

1. Manpower requirements and personnel resources data Inputs do have an 
influence on the equipment configuration, but this influence is attenuated by 
a complex of factors such as the engineer's indifference to and inability to 
interpret human factors considerations meaningfully; and more importantly, by 
the inadequate timing of PRD inputs. 

2. Tb<» potential influence of personnel requirements and PRD inputs on 
the equipment configuration is much greater than is presently achieved. PRD 
inputs would exercise much greater effect if they were: 

a. phrased as specific design requirements and constraints and in- 
cluded in the SOW (Statement of Work). 

b. phrased in concrete design-relevant terms. 

3. The manner in which the engineer designs has a significant effect 
upon his reaction to personnel requirements and his use of PRD inputs, hence 
on their influence on the subsystem configuration. The engineer's design 
concept is so quickly developed that traditional timing of manpower require- 
ments and PRD Inputs lag that concept. The engineer resists any change to 
his Initial concept as a restriction on his freedom to design creatively. 

4. Tlie importance of supplying meaningful manpower information as design 
requirements in the SOW and subsequent QQPRI analyses by trained human factors 
personnel, is highlighted by the fact that the engineer's manning concept for 
his own design does not appear to correspond to the needs of his subsystem 
designs. 

5. The results of this study are in accordance with those of previous 
design engineering studies. This Increases the confidence one can feel in 
the conclusions derived. 

CONCLUSION 

It was found that manpower requirements and PRD Inputs do influence the 
equipment configuration, but in this study only moderately, because the 
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equipment design proceeded so rapidly that incremental PRD inputs inevitably 
lagged the design. Engineers were responsive only to inputs which were 
framed as design requirements and which were interpreted in design-relevant 
terms. Confirming the results of previous studies, engineers were found to 
be generally unaware of or indifferent to personnel considerations. Differ- 
ent engineers interpreted the same design requirements and assigned priority 
to design criteria differently. The engineers relied heavily on experience 
and stereotyped solutions for design answers. The results of the study 
Indicate that manpower and personnel analyses must be performed prior to the 
issuance of a Request for Proposal (RFP) and not delegated to the develop- 
ment contractor. The contractor must be required to design to a detailed 
manning structure which is specified in his Statement of Work. Further 
recommendations are supplied which suggest ways in which Air Force management 
of the personnel subsystem program should be revised. 
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A-2 

Martino^ Joseph P. Technological Forecasting for Decisionmaking 



This book is intended to provide an advanced treatment of technological 
forecasting* It is a ''how to'' book: how to do technological forecasting, 
how to apply it in specific decision situations, and how to avoid some of the 
more common errors and difficulties in the preparation of individual forecasts. 
Although this abstract lists the chapter titles, it is primarily concerned 
with chapter 9, '^Normative Methods.'* 

GENERAL RESEARCH APPROACH 

The contents by chapter number are as follows: 



1. 


Introduction 


11* 


Research and Develop- 
ment Planning 


18. 


Teo huo lo^i i c.i I Fore- 
cast ing in Social 


2. 


Delphi 


12. 


Research 




Decisions 


3. 


Forecasting by Analogy 


13. 


Technology Advance*- 


19. 


Some Common Fore- 
casting Mistakes 


4. 


Growth Curves 




ment 


20. 


Dangers of Nega- 


5. 


Trend Extrapola- 
tion 


14. 


Product Development 




tive Forecasting 






15. 


Test and Evaluation 


21. 


Evaluating Fore- 


6. 


Analytical Methods 


16. 


Technological Fore- 




casts as Decision 
Information 


7. 


Breakthroughs 




casting in Business 
Decisions 


22. 


Presenting the 


8. 


Combining Forecast:^ 


17. 


Technological Fore- 




Forecast 


*9. 


Noirmative Methods 




casting in Government 
Decisions 


23. 


The Future of 
Technoloftical 


10. 


Planning and Decision- 
making 








Forecasting 



^Chapter 9 covered in this abstract. 



The objective of the normative methods is to determine the technological 
capability which will be required to carry, out some function, based on an 
estimated or projected demand. Exploratory and normative methods are not 
competitive with, or replacements for, one another. Both are essential, and 
both must be used together. Normally one does not bother to prepare an 
exploratory forecast of some technology unless there is a normative forecast 
(at least an implicit one) that the technology will be needed. Likewise, one 
does not normally prepare a normative forecast without some idea that it will 
be possible to meet the goals. 

ANALYSIS 

The three most common methods of normative forecasting are r elevance 
morphological models » and mission flow diagrams . Relevance trees are 
used where the system or process being analyzed can be described in terms of 
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levels of causation^ levels of complexity, or levels of hierarchy. Morpho- 
logical models are used where the system or process can be broken down into 
elements which can be altered independently > and where there are no hierarchi-^ 
cal relationships* Mission flow diagrams are used where the system or process 
can be described in terms of one or more paths of sequential steps. 

1. Relevance Trees 

Relevance trees are used to analyze systems or processes in which distinct 
levels of complexity of hierarchy can be identified* They are developed by 
carrying out successive identification of increasingly finer components, at 
progressively lower levels* Each branch divides (at a node) into two or more 
branches at the next lower level. Each node is a goal for all those branches 
depending from it. Each goal is satisfied by the achievement of all branches 
below it and, in turn, derives its validity as a subgoal from the sequence of 
branches linking it with the top of the tree. Relevance numbers can be as- 
signed to each of the branches of the tree and an overall figure of merit can 
he associated with a total path through the tree. 

2. Morphol ogical Models 

''Morphology" describes the study of the form or structure of the entity 
>f interest (i.e.* plants and animals in biology » rocks in geology). The 
essence of the morphological method of analysis is to break a problem down 
into parts which can to some extent be treated independently, with several 
solutions or approaches to each. A system or problem is broken down into 
parts which can be treated independently ♦ and as many soluti.is or approaches 
as possible are devised for each part. The technological performance re- 
quired for each solution is then obtained » and this is used as a goal for the 
technology involved. Some of the solutions can be rejected immediately on 
the grounds of feasibility; others can be rejected because of conflicts with 
other solutions. One is finally left with a set of feasible goals for the 
technologies involved. 

3 . M i^s s i on_F 1(^^^^ D i agrams 

The mission flow diagram was originally devised as a means of analyzing 
certain types of milUary missions. However, the t^echnique can be applied 
to the analysis of any sequential process. The technique involves mapping 
all the alternative routes or sequences by which some mission or task can 
be accomplished. All significant steps are identified on each route. It 
then becomes possible to identify the difficulties and costs associated with 
each routfi*. In addition^ it is possible to create new alternative routes, 
and identify the difficulties and coats associated with these. Once these 
difficulties and costs are identified, performance requirements can be de- 
rived for the technologies involved, and used as normative forecasts. 

FINDIN GS 

1. Normative, or goal-setting methods of forecasting are applied to 
determine the level of functional capability which must be achieved if some 
problem is to be solved or some difficulty overcome. 



57 

52 



2. Relevance trees are used to carry out analysis of hierarchical 
structures. 

3. Morphological models are used to analyze parallel structures. 

Mission flow diagrams are used to analyze processes with steps in 
a spatial, temporal, or logical sequence. 

5. Often the same system can be analyzed by either a relevance tree or 
a morphological model. In such cases, the forecaster should use whichever 
method is most convenient or appropriate for the problem at hand. 

CONCLUSION 

The strength of normative methods is that they tend to organize and 
structure the problem. They can help assure completeness, so that solutions 
holding some promise are not overlooked. By systematically laying out the 
structure of a problem, they can assist in the generation of new alternatives 
which may be superior to those currently in use. Even if all the alternatives 
uncovered with the use of normative methods prove inferior to those already 
known, this alone gives additional confidence in the choice of one of the 
methods as superior to the rest. 

It is essential that the normative methods be kept in proper perspective. 
They are in no sense a substitute for creativity or imagination.- They merely 
provide a syotematic means of examining technological requirements. 
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A- 3 

Hacke, James E. Jr., The Feasibility of Anticipating Economic and Social 

Consequences of a Major Technological Innovation 

Forecasting is methodologically distinct from goal formulation or 
planning and is as methodologically justified in its use of the results of 
empirical science as medicine or engineering. On analysis, the methods used 
for forecasting fall inti» two major classes, those for forecasting patterns 
and those for forecasting parameters , with five generically different methods 
available in each class. 

GENER AL RESEARCH APPROACH 

The discussion of each method includes description and examples » an 
investigation of the assumptions underlying its use, an estimate of its 
range of applicability, and a delineatiojiji Its limitations. These methods 
are applicable alike to technological, demographic, economic, and social 
forecast ing. 

Existing literature on past economic and social consequences of major 
technological innovations rarely go beyond narration and statistical descrip- 
tion; they fail to distinguish between changes following an innovation and 
changes resulting from an innovation. To make this distinction, one must 
delineate what would have been the course of events if the innovation had not 
been made. The methods of forecasting can be applied equally well to this 
delineation of hypothetical consequences. 

ANALYSIS 

The task of anticipating economic and social consequences of an innova- 
tion starts with forecasts of economic and social conditions over the period 
of time involved and forecasts in their light the probable level of effort 
likely to be expended in developing the innovation. Then one forecasts the 
probable pattern of its development and formulates and forecasts the changes 
In parameters descriptive of its performance. Having this description of the 
probable future technological development of the innovation, one can forecast 
probable economic and social impacts. Since these impacts modify the forecast 
economic and social conditions with which one starts, anticipating economic 
and social consequences of an innovation is in principle an iterative process. 

1. Technological forecasting was studied first under the assumption 
that methods of assessing economic and social impacts of innovations were 
much further developed than were those of technological forecasting. How- 
ever, the reverse proved to be the case. 

2. A survey and analysis of the literature on technological (and other) 
forecasting yielded ten generically different methods of forecasting tech- 
nological changes. 

3. A fundamental approach was to describe relevant portions of the 
economy and the society following introduction of the innovation and to 
describe what these might have been had the innovation not been introduced. 
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Then, one determined the difference between these two descriptions; the 
difference was a measure of impact of the innovation. 

FINDINGS 



1. Forecasting is as jusrifiable epistemologlcally as engineering, 
medicine, or any other discipline using the findings of empirical science. 

2. The methods of forecasting are basically the same whether applied 
to technology, demography, economics, or sociology. 

3. The division of forecasting into parameter forecasting and pattern 
forecasting runs through mathematics and science and is therefore probably 
fundamental. 



4. There seem to be five generically different methods of forecasting 
changes in patterns and five generically different methods of forecasting 
changes in parameters. 

5. Methods of forecasting cnanges in parameters include statistical 
measures that give some Vindication of the precision to be expected. When a 
parameter passes out of^4:he confidence limits set by these measures, however, 
the dynamics of the situation are changing and the forecast should be redrawn. 

6. The methods of forecasting changes in patterns or structures of 
organization lack precision or confidence measures and do not preclude 
alternative courses of development. This constitutes the single most impor- 
tant limitation to the current state-of-the-art in forecasting. 

7. The existing literature on assessment of the past economic and 
social impacts is of limited usefulness for the purposes of this study in 
that it falls into the post hoc ergo propter hoc fallacy. None of it com- 
pares what actually did happen with what might have happened had the innova- 
tion not come into existence. 

8. The largest single practical difficulty is likely to be locating 
adequate data for analysis. Data on the transistor through 1956 were not 
extensive enough to allow projection to 1366. In the absence of data over a 
long enough time span to establish trends, one must rely solely on informed 
judgment. 

9. The feasibility of anticipating economic and social consequences of 
a major technological innovation depends strongly on state of technological 
development of that innovation. Data are scarce in the early stages of 
developing an innovation, and there may be doubt as to what an innovation 
actually is and how it works. 

10. The specific forecasting techniques used are determined by the 
nature of the innovation. The transistor is a mass-produced electronic hard- 
ware component. Production and cost data were therefore necessary to fore- 
casting of its impact. A software innovation; a process innovation; a low 
volume, high cost item such as the laser; or a system such as interurban 
rapid transit would require other kinds of data. Availability of the 
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required data would determine feasibility of producing a meaningful anticipa- 
tion of future economic and social impacts. 

11. Although the "forecast" of the impact of the transistor and the 
forecasts suggested are useful as demonstrations, such forecasts would be 
of more value to planners if they were continuous processes. It would be 
particularly valuable to monitor a course of development that has been 
forecast to pinpoint when that course diverges from expectations and to 
obtain a more rapid check than is now possible on forecasting methods. 

CON CLUSI ON 

It is feasible to anticipate the economic and social consequences of a 

major technological innovation with sufficient detail and reliability to be 

of major assistance to the long-range planner. 

It is non feasible, however, to forecast quantitatively the trends in 
a parameter or a pattern, with the certainty that the outcome will fall 
within stated confidence limits. Such certainty would require both exact 
knowledge of the mathematical form of the trend curve and complete assurance 
that no perturbation of the trend will occur in the interim. 

Anticipating trends in the characteristics of a major technological 
innovation is feasible. It is feasible to anticipate some of them with 
enough precision and reliability for the forecast to be of significant value 
to the planner. The precision and reliability achievable as a function of 
time may match what the planner needs to know at that time. 

On the basis of these anticipated trends In the technical character- 
istics of an innovation and of its supply and cost, it is possible to antici- 
pate some of its economic and social consequences. In essence, this con- 
sists in comparing the probable future course of development In the economy 
and the society with what would be likely if the innovation had never 
appeared. Demographic, economic, and social forecasting are feasible to 
the same extent that technological forecasting is. 
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A-4 

Roberts, Edward B. Exploratory and Normative Technological Forecastin g; 

A Critical Appraisal 

(This report was contained as an appendix in the Marvin J. Cetron book, 
Technological Forecasting - A Practical Approach .) 

Increased recognition during the present decade of the importance of 
science and technology to corporate and national existence has produced an 
intensive search for new methods for managing research and development. The 
attention being devoted to so-called "technological forecasting" is one 
manifestation of this concern. 

GENERAL RESEARCH APPROACH 

This report presents a critical appraisal of the field of technological 
forecasting. The central theme is that the two phases of exploratory and 
normative approaches are out of step with each other. Exploratory techniques 
are too naive and do not take advantage of what has been learned about fore- 
casting in nontechnical areas. Nor do the exploratory techniques reflect 
what is known about the influences upon the generation of future technology. 
Normative techniques in contrast are too complex and mathematically sophisti- 
cated and cannot justify their elegance on substantive grounds. 

ANALYSIS 

"Technological forecasting," as defined by those claiming to be its 
practitioners, is actually tr^j fields, joined more by a vision than a reality. 
On the one hand is "exploratory forecasting," the attempt to predict the* 
technological state-of-the-art that will or might be in the future, or as 
Cetron puts it, "...a prediction with a level of confidence of a technical 
achievement in a given time frame with a specified level of support." Most 
laymen assume that all of "forecasting" is this kind of forecasting. The 
second aspect has been called "normative forecasting" and includes the organ- 
ized attempts to allocate on a rational bas-**? the money, manpower, and other 
resources that might affect the creation of tomorrow's technological state- 
of-the-art. Normative forecasting presumably provides aids to budgetary de- 
cisions in the technological area. Still more broadly defined by some of its 
leading exponents, "normative forecasting" applies to a wide variety of 
attempts to determine policies and decisions that will influence the effective 
growth of science and technology, in the corporation, the government agency, 
or the nation .is a whole. 

Exploratory technological forecasting includes a variety of techniques 
for predicting the future state of science and technology. Unfortunately, 
most of the methods are really only variants on simple trend extrapolation 
procedures, broadly defined, that have limited utility in today's rapidly 
shifting technological environment. The principal exploratory methods are: 

1. So-called "genius" forecasting, based either on individual wisdom 
or on a group "genius" forecasting process known as the Delphi technique. 
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2. Formal trend extrapolation to either a straight-line fit or an S- 
shaped expetrtation* 

The formal trend methods include single^curve projections as well as 
estimations based on the envelope encompassed by the projection of a family 
of related carves. They also include both the extrapolation of a single time 
series as well as the projection of ?ead-lag relationships between two time 
series* the latter known as "procursive event" forecasting. 

Normative technological forecasting activities attempt to provide a 
basis for allocating technology-generating resources so as to maximize 
attainment of organizational goals. The effectiveness of any normative 
method depends upon the following factors: 

1. Meaningfulness of its treatment of goals. 

2. Correctness of its assumed relationships between allocated resources 
and generated technology. 

3. Adequacy of its balancing of the resources-to-technology considera- 
tions against the worth of goal fulfillment. 

4. Impleraentability of the method^ including the ability to acquire 
reasonable inputs at reasonable costs as well as the ability to persuade or- 
gan ii^^ationa I decision-makers to use the generated outputs. 

Some of the leading normative forecasting techniques are: 

1. PROFILE (Programmed Functional Indices for Laboratory Evaluation) 

2* QUEST (Quantitative Utility Estimates for Science and Technology) 

3. PATTERN (Planning Assistance Through Technical Evaluation of Relevance 
Numbers) 

4. TRW*s PROBE II 
FINDINGS 

1. Comparison of the still evolving approaches to "exploratory" and 
"normative" technological forecasting yields marked contrasts. In particular, 
the simple schemes used by those trying to predict the technology of the future 
look pallid when matched against the sophisticated techniques designed by 
those who are allocating the resources that will create the futv^re^ Explora- 
tory technological forecasts are largely based either on aggregates of "genius" 
forecasts (e.g., the Delphi technique) or on the use of leading indicators 

and other simple trend-line approaches. The practitioners of economic fore- 
casting, in contrast, long ago recognized the need for multivariate systems 
analysis and cause-effect models to develop reliable predictions. 

2. Normative forecasting is at the opposite extreme on the sophistica- 
tion scale, fully utilizing Bayesian statistics, linear and dynamic programming, 
and other operations research tools. Here, despite the uniqueness, uncertainty, 
and lack of uniformity of research and development activities, each of the 
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designers of normative techniques has proposed a single format wholly quanti- 
tative method for resource allocation. Along the dimensions of unjustified 
standardization and needless complexity, for example, the proposed R&D alloca 
tion methods far exceed the general cost-effectiveness approach used by the 
Department of Defense in its program and system reviews. 

3. For both exploratory and normative purposes, dynamic models of broad 
technological areas seem worthy of further pursuit. In attempting to 
develop "pure predictions" the explicit recognition of causal mechanisms of- 
fered by this modeling approach seems highly desirable. This feature also has 
normative utility, provided that the dynamic models are limited in their 
application to the level of aggregate technological resource allocation and 
are not carried down to the level of detailed R&D project funding. 

CONCLUSION 

There Is no doubt that both kinds of "forecasting" are necessary con- 
tributors to the technical planning process, and for the military, as well 
as for most corporations, this technical input is a critical ingredient of 
an overall business plan. Yet only in theory but not in fact have these 
two components, the exploratory and the normative, been integrated adequately. 
Erich Jantsch expresses the logic of and the need for this integration: 

Exploratory technological forecasting starts from today's assured basis of 
knowledge and is oriented towards the future, while normative technological 
forecasting first assesses future goals, needs, desires, missions, etc., and 
works backward to the present .. .,^he full potential of technological forecast- 
ing is realised only where exploratory and normative components are joined 
in an iterative or, ultimately, in a feedback cycle". 
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A.3 

Halnrs, Donald B. Estimating Manning Requirements for Advanced «^vstems; 

f.ael» Sidru'v A Survey of the Defense Industry 

There is an urgent need for better use of our manpower. An important 
step is the early estimation of manpower requirements for advanced weapon 
systems. A simple and accurate method is needed for predicting these re- 
quirements as -arly as possible to prevent a subsequent surprising lew on 
available men and skills. 

ORNERAL RESE ARCH APPROACH 

This research was conducted in an effort to develop techniques of esti- 
mating manning requirements while the feasibility of a future system is being 
studied. One portion of this research program was aimed at Identifying and 
quantifying the relevant variables that generate manning requirements. One 
goal was to produce .1 multipurpose estimation technique. System designers 
could use such a technique to predict manning requirements during the early 
conceptual phase and trade-off manning requirements with other design con- 
straints and requirements. 

ANALYSIS 

The defense industry was surveyed to determine their methods and prob- 
lems oi estimatii.f? manning requirements. The survey had three principal aims: 

1. Determine the different ways the defense industry copes with the 
problem of estimating manning requirements for future systems. 

2. Assess t'.ie effectiveness of these methods if they were used with in- 
formation and data available only during the conceptual phase of the 
proposed weapon system, 

3. Decide which methods are more appropriate to specific kinds of 
weapon systems. 

The questionnaire used in the survey was a five-item, open-ended question 
type. Respondents were asked to give their name, field of specialization, job 
title, and the company for whom they were working. The following questions 
wt're asked r 

1. For which system (s) or suhsystero(s) have you participated in the 
estimation of manning requirements? 

2. Describe the technique you have used or would use to estimate the 
operator and/or the maintenance manning requirements for future 
systems. 

3. mich five factors do you consider to be important determiners of 
the number of men that will be required to support systems? 
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4. 



If you were asked to recommend three publications pertaining to the 
estimation of manninR requirements (either operntor or maintenam e) ♦ 
which ones would you Kti^gest? 



5. Additional comments* 



FINDINGS 



Questionnaires were sent to 117 people in key defense industries. 75 
questionnaires were returned (some completed by a team of respondents). Of 
these, 55 were fully completed and were answered in a way that was felt to meet 
the spirit of the survey. The principal findings of the survey are as follows: 

1. Dissatisfaction prevails; most industries feel that a new, unifom 
and standardized method is desired. At present there is no standard 
method that is applicable in the early stages of weapon system 
development. 



2. Predictions continue to be made so late that they are of little use 
in influencing design. Often a prediction is made in a form that 
does not lend itself readily to making design changes. 

3. There are definite trends and preferences in current use. Most re- 
spondents preferred traditional task-analytic methods for estimating 
manning requirements and were guided largely by USAF technical 
reports. 

4. The main differences were that the people who had access to acquisition- 
time data used historic comparison as an estimation technique, while 
those in the conceptual phase of development had more of a tendency 

to use other approaches, viz, modeling, 

5. Those respondents who had experience with space syst^s and communi- 
cations or radar nets used guides other than USAF publications; 
those with other system experience preferred the USAF guides. 

6. In general, the reliance upon USAF guides and materials was heavier 

in acquisition time. In other than acquisition time, USAF guides were 
used in conjunction with company reports. 

7. There was one significant finding from the statistical analysis: 
respondents who worked on space systems relied more on mathematical 
and objective methods of estimation than did those whose experience 
was with (communication or radar) nets. 

CONCLUSION 




Five different approaches to estimating manning were reported, the most 
popular being the use of task analysis or some combination of task analysis 
with variations of the other approaches. Other methods were: expert esti- 
mation, historical comparison, sovereign factors, and mathematical models. 
The most frequently cited guides were USAF publications, but a wide variety 
of other texts, handbooks, and technical publications were also reported. 
Respondent experience ranged over the major weapon systems, subsystems, and 
control networks. 

66 

61 



A-5 



Barton, H. R. , Purvis, R. E. , 
Stuart, J. E.. Mallory, W, K. 



A Queuing Model for Dctermtnl ng S ystem 
Manning and Related Support Requirements 



Increasing need for earlier estimates of manning, skills, and traininR 
requirements led lo the development of mathematically sophisticated techniques 
capable of computing and assessing requirements at every phase of system 
development. Current methods are largely Intuitive, rely c.» bookkeeping 
procedures, and are seldom applicable at pre-hardware stages of system 
development. The method in this report fills a need for making trade-offs 
when investigating alternatives in system design. 

CKNKRAI . RKSHARCH AP PROACH ' 

The report begins with an analysis of hardware functions and develops 
human requirements in terras of operational needs and service rates. Manning 
and skill requirements are integrated over such factors as desired environ- 
mental demands, maintenance concepts and procedures, and training require- 
ments. Two mathematical techniques, queuing theory and linear programming, 
are used to compute manning requirements and training needs. 

ANALYSIS 

The first step is the development of methods which enable engineers to 
compute and assess (at a very early time In the systems design) the manning 
and skill and training teqiiireroents for a given system design(s) with ac- 
companying operational and maintenance concepts. This report addresses 
three areas: 

1. Identify significant system variables that affect the requirements 
for manning, skillK. and training. 

The approach in this area was based on a mathematical model relating 
tsignif icant variaoles associated with manning and skill requirements; 

a. task identification which determines skill 

b. operational performance requirements 
The operational requirements were: 

(1) operational readiness specifieil for the system — a condition 
that must be satisfied. 

(2) capability of each operational unit to operate a total num- 
ber of hours or hours per mission for a certain number of 
missions per calendar time-unit- 

Q«i.infifv thi- I II format ion about these varlablfs. 

f'«»t oxampio: ()|»i-ral lonal rl^ldLMc•s^ (R) - .88 

total operational hours (to) " 16 
number .if missions (f) » 8 



mission hours (d) ■ 2 

number of equipments (N) « 18 



3- Develop mathematical techniques for relating these variables during 
the conceptual phase of system design and to compute > predict » and/or con-- 
trol the manning and skill requirements. 

Because of the need for hardware analysis and operational performance 
requirements > a causal mathematical model approach was pursued • The mathe- 
matical model relates: 

a, black boxes and associated repair and failure rates 

b» personnel 

c. spare black boxes 

d. waiting time to make a black box available aftt^r demand 

The output of the model (waiting time) is related to the system opera- 
tional requirements and» through application of an algorithm* allows the 
achievement of minimum manning for specified system operational requirements. 

In addition* a method was developed which will allow for scheduling of 
training-manpower resources in which consideration is taken of the following: 

a. training requirements in terms of time necessary to achieve a 
specified skill 

b. phasing- in of new systems and the concomitant d^anands on avail- 
able skills 

c* phasing-out of old systems and the concomitant availability of 
skills 

d. the manpower phasing into and out of personnel Inventory because 
of enlistment and discharge 

FINDINGS 

1. It is anticipated that the manning prediction technique in this re- 
port does not differ radically from intuitive procedures presently used by 
management personnel in the Air Force. The recognized differences lie in 
the formal structure of the manning problem* These differences are: 

a, mathematical statement of the manning goal 

b* the causal relationship between this goal and manning through 
the following: 

(1) spares 

(2) waiting time 

2, The manning objective in this report is mathematically equivalent to 
maximizing the rotal operational hours of the operational units ♦ given a fixed 
distribution of skill hours. 

ERJC „ 



3. The objectives are mathematically equivalent to raaxiniizing total 
training time (of operational units) for a given investment ip skill hours. 

A. Precise knowledge of operational requirements may not be known 
and it is desired to know how the manning varies as the operational re- 
quirements change. This ability to perform parametric analysis may be 
important in comparison of alternative systems or in jointly allocating 
fixed manning resources to two or more systems. 

5. The technique will allow evaluation of error in input Information 
through cause and effect relationships developed. 

6. Having established causal relationships, required refinement of 
specific prediction techniques may be directed. 

CONCLUSION 

Operating procedures must undergo testing and modification to achieve 
a good usable end product. For a prediction procedure such as the one given 
in this report, testing should examine various key characteristics such as 
reliability and validity. A reliable procedure should permit different 
people to achieve about the same results, given the same inputs. A valid 
procedure would have as its product, predicted manning close to that which 
actually is required to do the job. Potential error sources discussed in 
the report should be subjected to evaluation via a teat program for the 
manning technique. Analysis of causes of errors in predictions could lead 
to modification of procedures to remedy problem areas. 

Test application of the procedure would also provide a check on the 
clarity and comprehensiveness of the procedures and would point out areas 
where rewriting or revision is required for ease of application. 
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Purvis» R. E., Validation of Queuing Techniques for DetermtninR 

Mallory, W, K. System M anning and R^latPd Htipport Requirements 

McLaughlin, R. L. 

System ..inning has been a persistent problem with all industry and the 
military services. It has been a particular problem with the Air Force be- 
cause of the multitude of systems and personnel skill involved. The mathe- 
matical model to be validated here is based on finite cyclical queuing 
theory. Details of the model a^re contained in a previous report, "A Queuing 
Model for Determining System Manning and Related Support Requirements," 

GENERAL RESEARCH APPROACH 

A program was conducted to establish the validity and reliability of a 
technique of mathematical modeling for predicting manning requirements for 
weapon systems. The technique was applied to two systems: the F-105D fire 
control system (FCS) , which presently is operational, and the C-141 system, 
which is scheduled for operation in the near future, 

ANALYSIS 



Before a decision is made that a predictive mathematical model is 
acceptable, a validation is needed. The ways in which a prediction model 
may not meet stated objectives are: 

1. The model may include superfluous independent variables which have 
no appreciable effect on the dependent (output) variable. 

2. The model may omit significant independent variables. 

3. The model may assert an inaccurate, incorrect, or inadequate func- 
tional relation between the dependent variables. This may be the case for 
only certain ranges of values of one or more of the independent variables. 

4. A valid model apparently may fail due to inaccurate parameter pre- 
dictions. That is, inaccurately predicted values of an independent variable, 
substituted into the model, may yield an incorrect value for the dependent 
variable; whereas accurate predictions would have yielded an acceptable value. 

5. Every mathematical model of a real system is an approximation. An 
extremely accurate approximation is possible by taking many independent 
variables into account and using accurate values for these variables. If 
some of the variables are neglected because of poor model construction, or 
because they are not measurable, the approximation can be inaccurate. There- 
fore, it is necessary to decide in advance what degree of accuracy will lead 
to model acceptability. In general, sophistication and refinement of models 
are subject to diminishing returns, so that each further improvement bought 
at the same cost in technical effort will yield a smaller improvement in 
accuracy. Accordingly, it is desirable to attempt model sophistication only 
commensurate with desired accuracy. 
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The points discussed above form the fundamental criteria used in the 
validation of the model evaluated for manning prediction capability under- 
taken in this program. 

FINDINGS 

1. The manning technique evaluated in this program has been demon- 
strated to be a valid and reliable technique for predicting manning require- 
ments for weapons systems, 

2. The technique has been found to be relatively easy to apply > and is 
compatible with existing methods for predicting reliability and maintain- 
ability parameters* 

3. The model is based on random processes approximating field conditions » 
and produces valid results using state-of-^the-art reliability and maintain- 
ability techniques* 

4* The prediction for the FCS» using field data as sources for parameter 
estimation, yielded good results when compared with operational performance 
and exist Inp; manning* 

5* The results of the PCS analysis also verified the fact that» as with 
any technique, the validity of this technique depends upon the accuracy of 
the inputs. 

6, There is also evidence in the program results taat the technique pro^ 
vides a means for seeking out sources of inefficient personnel utilization 
and sparing procedures* 

?♦ It has been agreed that the reliability of a model lies in the 
ability to produce consistent results with different experimenters. 

8, The validation program did not formally test for reliability; how- 
ever^ a method was developed which establishes a general procedure for 
achieving reliability. This procedure was found to be quite sensitive to» 
and depc^ndent on, methods for estimating parameter values and organizational 
structure of the support system* 

CONCLUSIQN 

The model prediction for the PCS, using field data for parameter estima- 
tion, yielded good results when compared with operational performance. More- 
over, it was shown that the operational performance could be achieved by 34% 
fewer personnel than the manning set by the table of organization* The model 
prediction for the PCS, using conceptual data, resulted in substantially the 
same manning for the maintenance shop as that developed from the measured 
data; but, because maintenance concepts had been changed in the field, the 
number of flightline airmen was larger than the measured data* The manning 
prediction for the C-141 system, based on operational rates planned for the 
system and field data on the C-130 system, resulted in a prediction of 819 
airmen in the organizational maintenance squadron and 476 airmen in ield 
maintenance squadron* With the predicted manning it could be expect 1 that 
an operational readiness of 78% could be maintained. 
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Snyder, Melvin Techn iques for Developing Systems to Fit Manpower 

Askrent William B. Resources 

This report discusses four related processes for developins systems 
within manning and skills constraints. The point is made that the military 
services are beginning to feel the need to exert some measure of control over 
system design where human resources are involved. 

GENERAL RESEARCH APPROACH 

The report covers: (1) research to demonstrate that certain human re- 
sources data^ such as manning and skills data^ when used as design constraints 
along with other constraints does affect system design; (2) techniques by 
which to match or balance » through trade-off practices ♦ the hardware, the 
human and other system support or logistic functions in order to get a best 
mix depending on the real world operational goals; (3) a new specification 
by which personnel requirements are stated in terms of system requirements » 
for integrating more fully the human requirements with the system engineering 
processes; (4) a newly developed computer based data handling system for 
human resources data in the conception, development, test and operation of 
systems. The new data handling system is also an integrator of human and 
hardware data within and across systems, and a mechanism for comparing field 
performance data with early design data. 

ANALYSIS 

A study was conducted: 

1. to determine; the effect on system design of using certain human 
factors data, in this case, manpower and skills data as design requirements. 

2. to determine under what conditions these inputs could be made to have 
maximum influence on system configuration. 

The study simulated the design of the Titan III Propellant Transfer and 
Pressurization Subsystem (PTPS) by gathering the equipment and personnel 
documentation produced during the Titan III PTPS development and distilling 
them into the key inputs which has been provided to the original designers. 

Six highly experienced propellant system designers were selected as 
subjects and given the problem of designing the PTPS, stdrting from the 
Statement of Work. As equipment inputs, they received the information the 
original designers had received — functional requirements, equipment speci- 
fications, and hardware information. They also received in the Statement of 
Work a manpower constraint which required them not to exceed a certain crew 
size and skill level. in addition, they received incrementally other 
personnel resources inputs such as task sequences, human error data, and task 
time limits. 

During the test period, the engineer subjects were required to develop 
all the outputs they would ordinarily be required to produce in actual design: 
schematics, equipment descriptions, operating procedures, control panel 
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skoU'hes, lists of required hardware. The detail these engineers provided 
was great enough so that at the end of the testing period the subsystem de- 
signs rould be evaluated by mathematical systems effectiveness measures, 
including equipment reliability and safety. 

Iti additon, at the end of each test session, the engineers were inter- 
viewed intensively to determine whether in their opinion the personnel data 
inputs were useful to them and if the data had influenced their design. 

FINDINGS 

1. Manpower and personnel skill data can have an influence on system 
configurations. Examples of this influence are given in the report. The 
three designers with a design goal of a small crew with high skill produced 
top-level schematics averaging 4.7 human functions, and the three designers 
with a design goal of a large crew with low skill produced top-level schematics 
averaging 7.0 human functions. The "small crew" designers averaged 58.7 and 
the "large crew" designers averaged 68.0 for quantity of items of equipment 

to be operated by personnel. Finally, the "small crew" designers produced 
configurations that required on the average 6.0 personnel to provide opera- 
tion and maintenance, whereas the "large crew" configurations required 8.3 
personnel. 

2. The design approach followed by an individual engineer has a signifi- 
cant effect upon his reaction to the manpower and personnel skill data. The 
engineer's design concept is so quickly developed, based on his own past ex- 
perience, that traditional incremental timing of the human factors inputs 
lags that concept. The engineer resists any change to his initial design as 

a restriction on his creative freedom. 

3. The potential influence of manpower and personnel skill data inputs 
on system configuration is much greater than presently recognized or achieved. 
The inputs would exercise much greater effect if they were (a) Included in 
the Statement of Work as design contraints, and (b) phrased in concrete rele- 
vant terras. 

CONCLUSION 

This research has been successful. Several companies have studied this 
"pilot" system and have put similar processes into their own operations. The 
first purpose for this research has been attained — that was to develop and 
test computer techniques and design a "pilot" model for subsequent develop- 
ment into a data handling system. Based on this research, a proposed Advanced 
Development Objective, "Automated Information and Data Handling for System 
Program Offices" (AIDSPO) was prepared recently and is under consideration 
as a development program. 
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Meister, David, Rela tion ship Between System t>esipn. Technic^^ 

Finley, Dorothy L.» Tr aining and M.itntenan ce Job Performanc e on 

Thompson, Ernest A. Two Autopilot Subsystems 

Human resources requirements and data» particularly those related to 
training, have little significant impact upon equipment configuration. Having 
knowledge of equipment design, training, and experience factors doeu not 
necessarily provide information on how they influence performance. The study 
described In this report was aimed at providing Information on how these 
variables are interrelated. 

GENERAL RESEARCH APPROACH 

The study was designed to discover certain characteristics of equipment 
design which affect training time and perform/mce level. The specific pur- 
poses of the present study were to determine: 

1. Those design characteristics of the selected subsystems which the 
maintenance technician responds to in performing maintenance. 

2. How effective maintenance is on the selected subsystem. 

3. How technician effectiveness is related to particular design charac- 
teristics. 



4. What differences in maintenance perfoxrmance are related to differen- 
tial amounts of experience, training, and skill level of personnel. 

5. What parameters of skill maintenance personnel recognize as dif- 
ferentiating various levels of skill and how these relate to the 3, 5, 7 
level categories used in the Air Force. 

6. Whether the study methodology is effective and those modifications 
which should be made to improve the effectiveness of the study methodology. 

ANALYSIS 
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After the airborne electronic equipments in the inventory were reviewed 
two autopilot systems were selected for study. 

1. Subjects for the study were those technicians assigned to autopilot 
maintenance at two Air Force bases. 

2. The study methodology involved direct observations of technicians 
performing troubleshooting type maintenance. 

3. Following each maintenance incident, technicians were interviewed 
about the incident. 

4. At the conclusion of the study, maintenance supervisors ranked and 
rated all technician subjects on a series of specially constructed scales. 
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3. Data collectors also rated the major characteristics of the equip- 
ment maintained. These ratings were correlated with indices of technician 
performance. 

6. The resultant correlations were subjected to multiple regression 
analysis, and those factdrs contributing a significant amount to performance 
variance were extracted. 

As a possible substitute for prolonged observations, a subjective report 
test battery was developed to permit subject personnel to report their own 
maintenance performance. This battery was administered at the conclusion of 
the study to all technician subjects and to two classes of autopilot mainte- 
nance students. 

FINDINGS 

1. The performance data sought to determine the relationship between 
indices of maintenance behavior and various classes of predictor variables. 
After the variables which influenced maintenance performance were identified, 
they were organized into factors by interpreting their interrelationships. 
These factors are; 

a. Accessibility 

(1) to the equipment on which maintenance must be performed. 

(2) within the equipment to be maintained. 

b. Diagnostic information represented by 

(1) the information supplied directly from the equipment being 
maintained. 

(2) the information derived from test equipment used to check 
out the equipment. 

(3) information fromT.O.'s and checklists. 

c. Equipment structure consisting of 

(1) chassis weight and dimensions. 

(2) number, type, and arrangement of internal components, 

d. General technician capability, particularly for 

(1) test equipment usage. 

(2) mnintcnanre function and procedures. 

2. The data obtainc^d from the experimental test battery indicate that 
I'xptTitMK ed maintenance technicians can reliably predict certain aspects of 
m;! Int enance performance. 




3. Technicians can reliably estimate troubleshooting task times, task 
difficulty, and diagnostic strategy required by the maintenance problem. 



CONCLUSION 



Four general factors were found to be primarily responsible for the 
maintenance performance observed. The factors were accessibil^ diagnostic 
Information, equipment structure, and general technician capability. A 
residual experience factor was also found. From the results of the admini- 
stration of the subjective report test battery, it seems feasible for mainte- 
nance technicians to predict maintenance duration of failure problems with 
which they are familiar, the relative difficulty level of these problems, 
and the order In which they troubleshoot these failures. 

If the conclusions reached are to be accepted with any confidence, they 
must be replicated with a larger number of maintenance personnel and different 
equipments. Verification of the hypotheses raised by this study can be ac- 
complished by selecting equipments from the Air Force operational inventory 
which vary systematically in terms of the general factors found and then 
collecting maintenance performance data on those equipments, making use of 
the experimental test battery procedure. In addition to this, methods must 
be developed for obtaining better information on training parameters than 
current Air Training Command records provide. 
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Lintz, Larry M. , Relationships Between Design Characteristics 

Loy, Susan L*, of Avionics Subsystems and Training Cost , 

Hopper » Raymond, Training Difficulty, and Job Performance 
Potempa, Kenneth W* 

The human resources of the Air Force have a major impact on the 
operational capabilities and overall costs of systems* However, information 
on human resources requirements is generally introduced late or not at all 
into system design. Previous studies by the Air Force Human Resources 
Laboratory indicate such information does affect design if it is available 
in suitable quantitative form early in design and if the Statement of Work 
requires that it be used. This study investigated the relationship between 
avionics subsystem design characteristics and training time, training cost^ 
and job performance^ 

G ENERAL RESEARCH APPROACH 

A list of design variables believed to affect training and job perform-* 
ance wa.^ established and supplemented with personnel variables* Twenty-nine 
measures of the equipment design were taken, ranging from the number of test 
points to the number of throwaway components. The personnel variables used 
were aptitude test scores and amount of training and experience. Thirty 
functional loops, from ten avionics subsystems, were selected as the units 
of equipment to be Investigated. Functional checkout procedures were identi- 
fied or constructed for each functional loop. Ten Air Force (AF) students 
were observed performing each of the functional checkouts; performance time. 
Technical Order (T.O.) reading time^ and errors were recorded. Training 
time and training equipment cost data were collected for each loop. Regression 
analyses and factor analysis were used to analyze the results and derive 
equations to predict training time, training equipment cost, and job perform- 
ance time and errors from equipment characteristics. 

A NALYSIS 

A functional loop in this study is defined as a network of circuits and 
equipment units within an avionics subsystem through which signals are pro- 
cessed to perform a specific function ^ 

The principal requirements for a correlational study of equipment 
characteristics influencing maintainability can be summarized as follows: 

1. Establish a comprehensive and valid listing of design characteristics 
for inclusion in the study. 

2. Sample across classes of equipment so that the results of the study 
can be generalized to a range of practical equipment items. 

3. Collect as comprehensive a data base as possible using the best 
methods available. 

Thf KiMieral approach for this studv was designed with these criteria in mind. 
To meoL Lime and cost limitations in this probe, restrictions had to be 



Imposed, but these wore cho.sen to maximize the data base and the Ht^nerali/..!- 
bility. The items of equipment which were investigated included navigation, 
flight control, conununications, and fire control systems (30 functional loops 
within 10 different avionics subsystems). 

Thus far, the most satisfactory method of collecting maintenance perform- 
ance data is observation by observers who are familiar with the tasks tliey 
are scoring. The data base therefore, for this report, consisted of on-site 
real-time observations of performance by experienced data collectors. 

The major task was the functional checkout procedure since a complete 
functional checkout can be specified for each functional loop of the equip- 
ment, and the functional checkout is the first stage of troubleshooting. The 
checkout verifies that the reported malfunction is indeed present, that it 
is in the loop under investigation, and within that loop identifies the area 
of malfunction. 



The study was carried out in three broad phases: first, dotcrmiiu- tht- 
design characteristics to be included and select subsystems and functional 
loops to represent those characteristics.; second, collect data; and third, 
analyze results. 



FINDINGS 



1. Four prediction equations were developed through the use of stepwise 
regressions. 

2. Ninety-four per cent of the variance in maintenance task performance 
time can be accounted for by the complexity of the subsystem, the number of 
steps in the maintenance task, the reliability of the test equipment used, 
and the ruggedness of the components being repaired. 

3. Ninety-four per cent of the variance in task errors was found to be 
due to the number of steps in the ^airttenance task; the number of special 
conditions, such as cooling, that are required; extent to which the T.O. uses 
standard syrabology; the amount of training; the number of dependent romotoly 
located components; and the reliability of the test equipment. 

A. Eighty-one per cent of the variance in training time required for a 
task could be accounted for by the convenience of test point location, the 
length of the task, whether T.O. change data are clearly presented and the 
number of dependent remotely located components. 

5. Seventy-seven per cent of the variance in the cost of training equip- 
ment could be accounted for by the extent to which maintenance tasks are 
automated, the per cent of identical circuits used and the T.O. time. 

6. A factor analysis, using the orthogonal components method, resulted 
in identifying o factors which account for a major part of the variance in 
the 20 independent variables which correlate significantly with performance. 
Names assigned to the 6 factors are: 
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a. Length of Checkout Procedure 

b. Equipment Complexity 

c. Difficulty of Checkout Steps 

d. Nonautomatic Checkout 

e. Diagnostic Information 

f. Clarity of Information 

. 7. Stepwise regressions of the performance variables on the factors 
resulted in prediction equations with multiple correlation coefficients of 
0.92 or greater. 

CONCLUSION 

The magnitudes of the regression coefficients establish this as a very 
promising approach to deriving human resources data for systems under develop- 
ment. This has been shown to De true across a variety of avionics equipment, 
including communications sets, autopilots, air data computers, and fire con- 
trol systems. The degree of predictive accuracy attained in this study cannot 
be expected to remain as high during replications of this study. However, the 
results oi this study indicate that even wifh normal shrinkage the predictive 
capability of the equations will remain highly significant. 
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Lintx^ Larry M.^ Prj^dictinjj;^ M aint e^^^ ^^'^^}^ ^^fl^SWl^y. j ^ . ^^ Personne l 

f ,f>y » Susan L . » .^AU J. Re qaire ments Biisett o n j To si^n P/ir nm e t e r s of 

Potctnpa» Kenneth W. 

The human resources of the Air Force (AF) have a major impact on the 
operational capabilities and overall costs of systems. However, information 
on human resources requirements is generally introduced late or not at all 
into system design. Previous studies by the Air Force Human Resources Labora- 
tory indicate such information does affect design if it is available in suit- 
able quantitative form early in design and if the Statement of Work requires 
that it be used. This study investigates the relationship between avionics 
subsystem design characteristics and task difficulty » personnel skill char- 
acteristics, and job performance. 

GENERAL RESEARCH APPROACH 

The relationships among subsystem design characteristics, personnel 
skill characteristics, and job performance were investigated for avionics 
subsystems. A list of design characteristics was established, based on ex- 
pert opinions of avionics engineers. Air Force supervisors, and instructors. 
Functional loops and line replaceable units were selected from ten sub- 
systems representing navigation, flight control, communications, and fire con- 
trol subsystems. Experienced maintenance supervisors identified high skill 
and low skill maintenance personnel. The same supervisors associated per- 
formance times and error probabilities with these personnel for three mainte- 
nance tasks — an easy task, a difficult task, and a complete functional 
checkout task. Supervisors also rated each task on a scale of difficulty. 
Both stepwise regression and factor analysis were used to derive equations 
for predicting performance time, error probability, and design characteristics. 
Similar analyses derived equations relating performance time and errors to 
personnel characteristics. Personnel factors of Experience, Aptitude, Breadth 
of Skills, Motivation, Training, Time in Grade, and non-Air Force Technical 
Education were identified. 

ANALYSIS 

This study used a correlational approach to derive equations for pre- 
dicting performance times, errors, and task difficulties from equipment de- 
sign characteristics and personnel characteristics. Organizational level 
maintenance data were collected on 27 functional loops from 10 avionics sub- 
systems (a functional loop is defined as a network of circuits and equipment 
units within an avionics subsystem through which signals are processed to 
perform a specific function). Intermediate level maintenance data were col- 
lected on 28 line replaceable units (LRU*s) from the same 10 avionics sub- 
systems. 

Twenty-eight (for organizational level maintenance) and 29 (for inter- 
mediate level maintenance) equipment design characteristics, such as number 
of test points and accessibility of components, were measured objectively 
from the equipment itself or from the T.-hnical Order (T.O.). Where 
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c»!)jfi tivt' mcasuremimt was not possible, rating scales were completed by AF 
porsonnol. Sixtoon personnel variables, such as aptitude scores, amount of 
training, and military rank, were used. 

Three maintenance tasks were Identified for each functional loop and 
LRU — an easy task, a difficult task, and a functional checkout task. For 
each task, experienced AF maintenance supervisors estimated performance times 
and error probabilities for their high skill and their low skill personnel. 
Supervisors also rated each task on a scale of difficulty. 

The design, personnel, performance, and difficulty variables were tnter- 
correlated, and were entered into regression and factor analyses. Equations 
were derived to predict performance and task difficulty from design character- 
iKtics, and to predict performance from personnel variables and task diffi- 
culty. Personnel factors and design factors were identified. Personnel 
profiles were developed for high and low skill groups. Plots of performance 
versus difficulty and task completion versus time were developed. 

FINDINGS 

1. The multiple correlation coefficients (Rs> for predicting times 
from design characteristics -"anged from 0.87 for high skill personnel per- 
forming intermediate maintenance to 0.93 for low skill personnel performing 
intermediate maintenance, indicating that from 75 to 86% of the observed 
variance in performance times can be accounted for by as few as 2 or as many 
as 8 predictor variables such as number of steps In checkout procedure, 
accessibility of components, use of identical circuits, and level to which 
checkout is carried (LRU, module, or component). The Rs for the error 
equations ranged from 0.59 for high skill personnel performing organizational 
maintenance to 0.85 for low skill organizational maintenance, with 1 to 4 
predictor variables accounting for 36 to 72% of the observed variance in 
error probabilities. 

2. The equations for predicting task difficulty of functional checkout 
procedures from equipment design characteristics gave Rs of 0.73 at organiza- 
tional level (53% of the variance) and 0.80 at Intermediate level (64% of 
the variance), from 4 predictors. 

3. Regression equations for predicting performance times from task 
difficulty and 1 or 2 personnel characteristics yielded Rs of 0.50 for high 
skill intermediate maintenance to 0.73 for high skill organizational mainte- 
nance (25 to 53% of the variance). The corresponding equations for error 
probabilities from 3 to 5 predictors yielded Rs from 0.59 for high skill 
organizational maintenance to 0.78 for low skill organizational maintenance 
(36 It) 62X of the variance), 

4. Factor analyseK on the performance and personnel variables isolated 
(actors of i.cn^Uii of Service, Aptitude, Experience, Motivation, Air Force 
TiThnic.il TraiMin^'., and Time in (Jrado, at Llio organizational level. At the 
IntL-rmudiatt' level, the factors were Length of Service, Aptitude, Electronic 
Aptitude, Avionics Experience, and non-Air Force Technical Training, 

5. Factor analyses of design variables indicated that factors associated 
with m;iintenance performance times and error probabilities were Checkout 

81 



ERIC 



Complexity » Checkout Information, Length of Checkout, Accessibility, Equip- 
ment Complexity, and Test Equipment and Adjup«^ments. 

6. Personnel profiles for the 16 personnel variables v/ero developed 
lor each subsystem and for each maintenance level; at orRanizat ional and at 
intermediate maintenance levels the low skill profiles have approximately 6 
times the error probability of the high skill profiles. 

7. Plots of time and errors as functions of difficulty level indicate 
accelerated relationships. High and low skills are more clearly separated in 
errors than in performance times. 

CONCLUSIONS 

The results of the present study should be replicated and extended across 
additional subsystems and types of avionics equipment. The performance 
measurement methods should be validated, and a better technique for deter- 
mining error probability should be developed. The relationship between task 
difficulty and the performance measures should be further investigated and 
clarified. 
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A-12 

Grossman. Edward R.F.W. . The Impact of Technological Change on Manpower 
Lancr, Stephen and Skill Demand; Case-Study Data and Policy 

Implications 

The research in this report represents an expansion and development of 
earlier work by the present authors reported in October 1966 under the title 
"Evaluation of Changes in Skill Profile and Job Content Due to Technological 
Change: Methodology and Pilot Results from the Banking, Steel and Aerospace 
Industries." 

GENERAL RESEARCH APPROACH 

The overall objective of this report was to analyze manpower and skill 
impa-t of technological change. The specific objectives werft: 

1. Apply the skill-profile methodolojiY developed in the course of 
previously re.'^^-Jtd ^ ..ot investigations to direct labor in further types of 
technologi- ; t chereby enlarging the available data base and permitting 
more; positive: v -;. rj.»i;; ons to be reached. 

2. Exv. • ' i . ' _thodology and/or modify it to obtain parallel data for 
indirect labor . . a j?^";aet of the processes studied. 

3. Provide an overview of the manpower demand pattern indicated by the 
data and draw conclusions relative to manpower policy. 

ANALYSIS 

1. Outline of the "Skill Profile" method 

The "Skill Profile" is the distribution of total man-hours required to 
produce a unit product (or service) along the scale from least to most highly 
skilled labor. A suitable basis for the construction of such skill scales 
has been established using skill factor scores derived from job evaluation 
schemes used in most industries. Comparison between skill profiles of newer 
and older processes permits statistically controlled conclusions to be drawn 
concerning the impact of technological change on manpower and skill require- 
ments, provided that precautions are taken to control the effects of the 
following factors: 

a. processes to be compared must be matched for the nature and 
quality of their material inputs and outputs. 

b. they must be in a steady state to avoid contamination by 
transiently increased man-hour and/or skill requirements due to 
process development, debugging, etc. 

c. organizational differences between firms must be controlled by 
replicating each comparison in at least two firms. 

Paired comparisons of the manpower and skill demand pattern before and 
after a specific technological advance can be made fully quantitative by the 
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statistical technique of analysis of variance. The effect of central in- 
terest in the present study was the (skill level x technology) interaction 
which was tested for significance against the residual variance in the (skill 
level X technology x firm) interaction term. When the variance ratio was 
significant at a selected confidence level, it was inferred that the observed 
change in skill profile had been caused by the technological change studied. 

2. Selection of further direct-labor case studies 

The skill profile method previously developed was applied in further 
direct-labor studies on technological changes in electricity generation, oil 
refineries, air separation, and airline passenger reservation systems. As 
before, these cases were each selected as being typical of a large class of 
similar technological changes in the same and other industries. In each case 
a fairly recent and pronounced technological advance had become stabilized 
and was deemed likely to replace the older technology in the near future. 

3. Extension of the "Skill-Profile" method to indirect labor 

A criterion of "Process Relatedness" was adopted to determine or identify 
the indirect labor force. Included were those functions whose withdrawal 
would cause a rapid decay in process performance. Indirect labor was traced 
and allocated to unit production with the inclusion of maintenance and first 
level supervision, planning and scheduling, and the exclusion of technical 
and research personnel and higher management. 

Although difficulties were encountered in acquiring data for Indirect 
labor skill profiles, the feasibility of applying the skill profile method 
was established, and complete (direct plus indirect labor) profiles were 
obtained for a sizeable subset of the total case-study complement. 

FINDINGS 



1. Labor Productivity 
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a. T.abor productivity increased in 72% of cases, and remained the 
same in 28%; direct labor productivity was not reduced by any of 
the technological changes. 

b. Greatest gains were recorded where the prechange technology re- 
quired the greatest amount of direct human intervention in the 
process. 

c. In most cases (88%) inclusion of indirect labor data reduced the 
productivity gains computed from direct labor alone. 

d. There was substantial variance between organizations due to 
identical (or near identical) technological changes. 

2. Changes in Skill Profile 

t 

a. The limited amount of indirect-labor data that could be acquired 
within the resources did not permit statistically controlled 
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coinparisons between pre- and post-change skill profiles for the 
total process related work force* 

b» Mean direct labor skill levels increased in 50% of comparisons » 
remained unchanged in 33%, and declined in 17%. 

c. Direct labor data results support the view that technological 
advance tends to increase skill demand • 

d. On Che average, technological change tends to decrease the skill 
demanded on the process-related work force in manufacturing and 
service industry. 

Skill increases observed in the direct labor force were largely 
cancelled out by decreases at indirect labor level. 

f . Increases in overall skill level are associated with technologi- 
cal advance only in the service sector, where labor demands are 
relatively low. 

3» Impact of Technological Change on Work Force Educational Levels 

With only one exception^ the newer technology required a better educated 
labor force than the old. The general nature of the process had more effect 
than the technological change in itself • 

CONCLUSION 

It has become clear from the studies completed on a pilot basis chat 
direct labor data taken alone yield a misleading picture of the overall 
manpower and skill impact of a given technological change. It is necessary 
to emphasize conclusions based on total data where these are available, even 
though for Lick of replication they were not tested for statistical signlfi- 
canre* Further research effort would be needed to round out and substantiate 
the present conclusions by completing the 18 comparisons undertaken. 

Other factors were not fully investigated within this report. Of these, 
the most important is probably the relative effectiveness of manpower 
util ligation across processes and technologies. 
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A-13 

Askren, William B. , Design Option Decision Trees; A Method for 

Korkan, Kenneth D. Relating Human Resources Data to Design Alternatives 

The human resources of the Air Force have a major impact on the opera- 
tional capabilities and overall costs of aerospace systems. One of the re- 
search objectives of the Air Force Human Resources Laboratory is to develop 
methods of incorporating human resources data (HRD) in the decisions which 
define systeri design characteristics. This study was performed to determine 
the feasibility of predetermining the design options available to the engi- 
neer as he progresses through a design problem. 

GENERAL RESEARCH APPROACH 

It would be most useful if a method could be developed for relating human 
resources data to detailed design features. These data could then enter into 
decisions regarding selection of specific design options. The purpose of this 
investigation was to determine the feasibility of identifying the design 
options available to the engineer as he progresses through a design problem. 
The approach involved three steps: 

1. Design options available to the engineer in the propulsion and 
flight control subsystem areas were identified and placed in decision tree 
form. 

2. Design options and the concept of arranging the options in decision 
tree form were evaluated by eight engineers experienced in design of these 
classes of subsystems. 

3. The information and data obtained from the engineer evaluators were 
analyzed (after personal Interviews) . and new decision trees of the propulsion 
and flight control areas were prepared to represent a composite of their 
recommendations. 

ANALYSIS 

Two experienced engineers worked together to produce five first genera- 
tion decision trees describing propulsion and flight control subsystems. 
Eight engineers were chosen as evaluators; four with experience in propulsion, 
and four with experience in flight control. Each evaluator was given a set 
of the decision tree drawings related to his area of experience and asked to 
study and critique them. After a few days, an interview was held with each 
of the evaluators. During the interview the engineer was questioned con- 
cerning his opinion of the validity of the design options and the flow and 
nature of decisions represented in the tree. He was also asked for his 
evaluation of the completeness of the tree with respect to the design options 
available in the subsystem area, and the feasibility and practicality of 
representing the available options in a decision tree format. And finally, 
he was asked to recommend changes to the tree in order to represent the de- 
sign decision procass according to his experience. The information and data 
Obtained from the evaluators were analyzed, and new decision trees of the 
propulsion and flight control areas were prepared to represent a composite of 
their recommendations. 



ERIC 



86 

81 



FINDINCS 



1. Feasibility and Practicality of Preparing Design Option Decision 
Trees 

It was found to be feasible and practical to prepare design option 
decision trees, to identify the laajor disciplines or subsystems of an aero- 
nautical system, and to further determine the delineation of design options 
relevant to the subsystems. 

2. Validity and Thoroughness of Design Option Decision Trees 

All eight evaluators Judged that the originally prepared decision 
trees were valid representations of the design decision process, and, although 
not complete, adequately demonstrated the concept. The second generation 
trees were developed after recommended expansions and changes were given by 
the evaluators. Although design option decision trees can be made initially 
utilizing one or two people, it is necessary and feasible to Incorporate the 
consensus of a number of experienced engineers to develop the final concepts. 

3. Relating Human Resources Data to Design Options 

All eight evaluators judged that the design options should be meaning- 
fully evaluated for human resources factors such as maintenance difficulty, 
personnel skill, training difficulty and manpower costs, as well as for 
engineering parameters such as reliability, development costs, performance, 
and weight. 

4. Computer Processing of Design Option Decision Trees 

Recently developed computer software allows the storage and retrieval of 
tree forms of data. It is therefore highly likely that design option decision 
trees can be processed by computer. Computer software must be expanded to 
include the capability of storing the human resources implications of the 
design options. This would allow the design engineer to work throu^ih a de- 
sign problem at the computer console with computer memory providing the human 
resources (or engineering) implications of the design alternatives under 
consideration. 

5. Other Uses of Design Option Decision Trees 

A number of other applications of the decision tree forinat for design 
Information were identified: 

a. The decision tree format clearly indicates the number of choices 
available to the engineer at each decision node. 

b. It shows the interrelationships of decisions. 

c. The format allows for describing advancements to the state-of- 
the-art at the decision points. 

d. It provides a priority sequence of the decisions. 
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CONCLUSION 



The preparation of design option decision trees for propulsion and 
flight control subsystems was found to be feasible and practical. Although 
design option decision trees can be made initially utilizing one or two 
people, it is necessary and feasible to incorporate the consensus of a 
number of experienced engineers to develop the final concepts. Design options 
can be meaningf u7 y evaluated for human resources factors such as maintenance 
difficulty, personnel skill, training difficulty and manpower costs, as well 
as for engineering parameters such as reliability, development costs, per- 
formance, and weight. It was also determined that the design option decision 
trees can be processed by computer. This would allow machine storage and 
retrieval of the many design options and the related human resources data 
which could be used to evaluate each option. 
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Askren» William B 
Korkan^ Kenneth D 
Watts, George D. 



Human Resources Sensitivity to System Deatgn 
Trade-off Alternatives; Feasibility Test with 
Jet Engine Data 



Historically, skilled maintenance personnel were provided for Air Force 
weapon systems after the hardware was delivered to the operational command. 
Maintenance personnel selection and training was accomplished as a form of 
reaction to the demands of the equipment » Today, because of system complexity 
and the faster pace of events, human resources planning is characterized by 
analyses which predict maintenance manpower needs early in system development 
life. This allows personnel selection and training to begin before the hard- 
w'are is (delivered to the field. 

GENERAL RESEARCH APPROACH 



The feasibility of developing Design Option Decision Trees to a level of 
detail which shows hardware involved in **on-aircraf t" maintenance, and the 
feasibility of measuring the sensitivity of human resources data to design 
trade-off problems depicted in these trees were investigated. The approach 
included expanding a portion of an earlier developed aircraft jet engine tree, 
selecting trade-off problems from the expanded tree for sensitivity analysis, 
and collecting psychometric data from experienced jet engine mechanics re- 
garding this sensitivity. Five Design Option Decision Trees were devel'^ped 
for turbofan jet engines. Eight turbofan trade-off problems were evaluated 
for effect on human resources. The factors of training and experience » and 
amount of troubleshooting time (as affected by choice of design option); ease 
of maintenance; complexity of tools and equipment; and crew size and job 
specialty were examined. 



1. A portion of the aircraft propulsion Design Option Decision Tree 
developed in the earlier study was selected for expansion. 

2. An attempt was made to develop turbofan jet engine Design Option 
Decision Trees down to a level of detail depicting the smallest item of hard- 
ware replaced during **on-aircraf t*' maintenance. This i. comparable to the 
maintenance four-digit work unit. 

3. A number of design option trade-off problems representing a variety 
of jet engine characteristics were selected for sensitivity analysis as to 
their effect on Human Resources Data. 



4. Psychometric data were collected from experienced jet engine mechanics 
as a means of measuring the human resources sensitivity. Data were obtained 
regarding the effect of the design options on maintenance crew size, mainte- 
nance job specialties, training and experience, troubleshooting time, ease of 
maintenance, and complexity of tools and equipment. 



1. It was found feasible and practical to prepare Design Option Decision 



ANALYSIS 



FINDINGS 



Trees detailed to the maintenance four-digit work unit level of detail for 
turbofan jet engines. 

2 Five trees were developed and include a top-level drawing that shows 
the relationship of turbofan jet engines to the propulsion subsystem. 

3. Four trees were developed that describe the design options for jet 
engine assembly, jet engine installation, and jet engine supporting systems. 

^ , ^* levels of detail in the trees are sufficiently varied (such as 

choice of type of fan, choice of type of starter system, choice of type of 
compressor bearings) to provide trade-off problems with a variety of hardware 
characteristics. 

5. Eight trade-off problems were selected for study regarding effect 
on human resources. Six experienced jet engine maintenance mechanics eval- 
uated the design options: 

a. The results show that the amount of training and experience re- 
quired for troubleshooting a failure is significantly affected 
by choice of design options in six of the eight problems, with 
as much as 60% less training and experle ice being required for 
the favored options. 

b. Amount of time required to troubleshoot a failure Is signifi- 
cantly affected by choice of design options in six of eight 
problems, with as much as 56% less troubleshooting time required 
for the favored options. 

c. The complexity of tools and equipment required to troubleshoot a 
failure is affected in two of eight problems. 

d. Overall ease of maintenance is significantly affected by choice 
of design optirn in five of eight problems. 

e. Crew size and job specialty required to troubleshoot a failure 
are each affected in one problem. 

off t' J^^ «=o°>bined results indicate that at the start of a subsystem design 
effort the required design trade-off problems can be identified and evaluated 
for impact on the human resources. "^uaueu 

CONCLUSION 

T.. i« concluded that it is feasible to develop Design Option Decision 
of H^r^n ^^"^-'^isit work unit (line replaceable unit) level 

1 t h t P^-^viding a description of the broad spectrum of trade-offs 
if Lr^?M t Pf ^"'^^S ^ subsystem design. It is concluded that it 

is feasible to determine the sensitivity of human resources data to trade-off 
problems and design options using psychometric analysis methods. It is con- 
cluded that selection of a design option will influence numan 'sources and 
maintenance factors for a substantial proportion of design trade-off studies 

co:::id^; it ii iz^'''^' ^ '^^'^ ^^^^^^^ ^^^^ --^^^ optionfto 

consider it is highly probable that an order of priority of design options 
based on human resources effec «.an be determined. options 
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A-15 

Thomas. Ralph E., The Effect of Various Levels of Automation on 

Prltsker, A. Alan, Human Operators' Performance In Man-M achine 

Chrlstner, Charlotte A., Systems 
Byers» Richar *. H. 

Many systems must be monitored at regular intervals and maintained within 
specified tolerance limits. A pilot, for example, must monitor the speed and 
position of his aircraft and keep within some limits of heading and altitude. 
The necessity for controlling these system processes arises because they are 
subject to disturbances both erratic and constant. In other words, the 
system process is subject to a noise component superimposed on a signal which, 
on the average, drifts out of the specified tolerance. 

GENERAL RESEARC H APPROACH 

This report describes a method for generating definitive data on the 
effects of various levels of automation on human operators' performf.nce In 
man-machine systems. The method incorporates a model and equipment for 
theoretical and experimental investigations. Equipment was designed and uuilt 
in accordance with the assumptions of the automation model for studying human 
performance in an automation environment. Functions to be controlled are 
generated by a general-purpose analog computer. 

ANALYSIS 

The conceptual model which is used contains three essential elements: 
the noise parameter, the drift parameter, and the tolerance limits for the 
variable to be controlled. In addition, the model contains two essential 
human parameters: the monitoring frequency and the subjective tolerances. 
These subjecrive tolerances are perceptual limits which may or may not be the 
same as the actual tolerances. They are included to permit the human to 
correct the system whenever he thinks it is necessary. He may, for example, 
make a corrective action based on his subjective estimate when the system 
actually is not out of control. These subjective tolerances form the speci- 
fically human inputs to the model, are variable from subject to subject, and 
vary from time to time in a given subject. They are also subject to change 
due to motivation, drive, competition or cooperation, fear, fatigue, and 
other complex psychological factors. In addition, these tolerances are sub- 
ject to change by the degree of uncertainty the subject feels about his 
ability to control the system. 

The model allows the designer to incorporate human performance properties 
into the control subsystem by providing methods to determine the following 
relationships: first, how noise and drift are associated with subjective 
tolerances, and second, how complex psychological factors are associated 
with these tolerances. Knowing these relationships, the designer can modify 
his equipment to conform to human requirements, or personnel can be selected 
on the basis of the adequacy of Ll.viir subjective tolerances to control a 
specific system. 

In the model, the tasks of the human are repetitive cycles of three types 
of assessments for the control of the system. He must determine whether or 



91 86 



not the system is in control, select the corrective action to restore or 
maintain control, and judge the effect of the corrective action. The analysi^ 
is thus reduced to manageable proportions by focusing on the correctness of 
the assessments and the manner in which they are made under various conditions 



The conceptual model can be used to describe almost .-,11 man-machine 
systems These systems have at least four elements. First, they have some 
tF-Ia ,1 Second, these systons have a capacity to process information. 

iMird, all of these systems are expected to perform some task; that is to 
say, they have an output. The fourth element is provision for control of the 
system. Each system has criteria of performance associated with It The 
control subsystem takes the kind of action necessary to enhance the'attain- 
ment of the performance criteria. 



Functions to be controlled are generated by a general-purpose analog 
computer. Automatic recording of input and response signals for all three 
assessmeit types is accomplished on a six-channel recorder. In addition to 
having separable panels to permit future studies of different organizations 
of groups of human operators, the equipment may be preprogrammed to simulate 
lure by making certain correction controls "inactive" at specified times. 



failur 
ANALYSIS 



1. The critical experiments showed that the equipment is suitable for 
conducting automation experiments. In addition, the effects of parameter 
variations of the stochastic model yielded results generally as expected. 

2. Three pilot experiments were conducted that involved changes in the 
parameters of the stochastic model, the level of automation, the subjects' 
tasK, ana the assessment cycle time. 

3. For each experiment,, a description of a hypothetical, but realistic 
problem was given to the subject. Preliminary training sessions were con- * 
ov^or?J^ automation environment to the subject, to explain the 
experiment, to describe the equipment and its use. and to specify the task to 
be performed. At the completion of each experiment, a questionnaire concern- 
ing the experiment was answered by the subjects. 

4. In Experiment 1, the subject could not predict the behavior of the 

^^^^"^^ ^ ^^^S^ component included in the 

Ln^^r% . ^^T.' rules adopted by the subjects did not 

appear to involve prediction. Because the decision rules programmed into 
and ZTiZ ±? """'"^"^ prediction, no aigniflcant differences Letween human 
and machine performance were detectable for the various levels of automation. 

in nr^;. f P^^l^^ent 2, the parameters of the stochastic model were changed 
Jhe levels of"rr%?'' f edlctability of the relevant variable. Moreove^. 
n automation for the Type I (is the system in control?) and Typ^ 
^LirL ? ^°^^«^tive action to restore or maintain control) assessments were 
altered in such a manner that the overall level of automation was not changed; 

TeTels of ~lr' performance scores among the various ' ' 

xeveis or automation were observed. 
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6» In Experiment 3» two of the extreme levels of automation used in 
Experiment 1 and the stochastic model of Experiment 2 were U9ed» and a 
significant difference in performance scores was observed* A quantitative 
comparison of the theoretical and observed results for Experiment 3 could 
not be made because the magnitudes of the corrective actions were not suit- 
ably chosen in this experiment* However ♦ qualitative agreement with the 
model was fully demonstrated. 

CONCLUSION 

The efforts in this research program have produced the following tangible 
results: 

1. A qualitative model in which the automation problem is reduced to its 
essential features* 

2. A quantitative model that includes two system parameters ^ and may 

be analyzed to determine the assessment cycle times that will yield a desired 
probability of control for given tolerance limits p and specified magnitudes 
of the corrective actions* 

3. An experimental apparatus that is capable of presenting to groups of 
subjects a wide variety of control problems characterized by the specific 
values used* 

4. The results of pilot experimental studies that suggest satisfactory 
agreement between the predicted and observed probabilities of control. 
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A-16 



Baechner^ William R» Technological Change and the Qccupatlonal Composition 

of the American Labor Force, 1950 - 1960 

Technological change is one of the most important factors affecting the 
American labor force , but as yet one of the least understood* The kinds of 
models used by economists to examine technological change have been among the 
primary sources of this lack of understanding; they tend to focus on two-factor 
nK>dels and examine questions concerning, e^g*, whether or not there is a 
labor-saving bias connected with current technological change* This report 
examines the effects of technological change on the occupational composition 
of the U.S. labor force between 1950 and 1960 using an input-output model, 
with each industry's labor input coefficient disaggregated into 240 occupa^- 
tlonal coefficients. 

GENERAL RESEARCH APPROACH 

This report utilizes an input-output model to measure the effects of 
changes in: 

!• The level and composition of final demand 

2. The input-output coefficients 

3» Labor productivity In each industry 

4, The occupational coefficients in each industry on the economy's 
total employment by occupation. 

The model is used to measure the effects of technological change on the 
kinds of jobs performed, the level of complexity^ and their educational and 
training requirements* The model is used to determine whether the occupa- 
tional substitutions which occurred as a result of technological change 
were affected by changes in the relative wages of the different occupations. 

ANALYSIS 

Three questions were confronted: 

1, How did technological change between 1950 and 1960 affect the 
economy's demand for labor from individual occupations? Also, given the fact 
that technological change can work to alter a number of different kinds of 
labor-demand variables, which facets of technological change were most im- 
portant in determining occupational employment trends? 

2, Is it important for manpower planning to ask how technological change 
affected the kinds of work performed in the economy's jobs, the content level 
of that work, and the educational and training requirements? 

3, Did the wage changes between 1950 and 1960 induce factor-^substitu-* 
tion through technological change or was technological change autonomous? 
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A model was made* to answer these questions • The model was a modified 
input-output model developed by Wassily Leontief . It was assumed that the 
production functions for the economy's industries are of the fixed-input- 
coefficient form, with not only intermediate inputs, but labor inputs as 
weU» proportionately related to output levels. The model was implemented 
with input-'Output data from the 1947 and 1958 input-output studies and with 
employment by occupation by industry matrices from the 1950 and 1960 Census 
of P opulation. Adjustments were made to the original data to correct for 
the different time periods used and for different industry categories used 
in compiling and reporting the data. The final model was implemented using 
65 industries and 2hO occupations. 

1. Each of the possible sources of change had much different effects on 
labor deirind, totally as well as on indivl v .rl occupations. 

2. Final demand changes worked to increase the demand for labor from 
all but four occupations; the average unweighted increase was 33.6 per cent 
with very little variation among occupations. 

3. Technological change worked to reduce labor demand from most occupa- 
tions> but its effect was very diverse, 

U. Most importantly for manpower planning » it appears that the main 
source of variation in the growth rates of employment by occupations is to be 
found in the changes which individual industries made in the occupational mix 
of their labor inputs. 

5. Variation in occupational demand was due to technological change » 
and more specif icallyt to occupational mix changes. 

6. Changes in the final demand vector were not significant factors in 
altering the pattern of work performed in the economy; the only jobs which 
experienced much faster than average rates of growth from this variable were 
those involved with work in the fields of health* education* welfare, and 
research and design. 

7. Technological change* however* did measurably affect the kinds of 
work performed* reducing employment in jobs associated with tools and machines 
while increasing demand for labor in service-oriented jobs. 

8. Final demand and technological changes were both important factors 
in raising the overall content of the economy ^s jobs; both worked to Increase 
the economy *s demands for jobs involving high content levels and to decrease 
demand for low content jobs. 

9. Technological change over the decade tended to increase the amount 
of education required for jobs* by increasing demand for occupations needinj', 
a high school education or more and decreasing demand for Lhos*» requir lt)j; less 
than 12 years. There was no discernible effect on job training requirements. 

10. Final demand changes had effects on educational and training re- 
quirements which were similar to those of technological change. 
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11. Technological change was generally factor-biased, vith most in- 
dustries substituting rising wage occupations for falling wage occupations • 

12. Anticipated wage changes were a significant variable in the choice 
of technology > but accounted for very little of the variation among occupa- 
tional substitutions. 

CONCLUSION 

It is possible to use an input--output model to examine the effects of 
technological change on the occupational composition of the labor force. 
These changes include level and composition of final demand, input-output 
coefficients, labor productivity in each industry, and the occupational 
coefficients in each industry on the economy's total employment by occupation. 

The degree of accuracy with which these changes can be measured is not, 
however > determined. 
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A-17 

Grossman, Edward R. F. W. . Evaluation of Changes In Sktll-»Prof nnH 
Laner, Stephen, Job-Content Due to Technological Chance: 

Davis, Louis E., Methodology and Pilot Results From the Banking. 

Caplan, Stanley H. Steel and Aerospace Industries 

A new method has been developed for measuring the impact of selected 
advances in technology on the distribution of skills required in the direct 
labor force in manufacturinp, and service industries. Based broadly on the 
direct productivity" studies conducted by the Bureau of Labor Statistics 
m the late forties and early fifties, the procedure as applied to a single 
process entails evaluating the man-hours of direct labor expended per unit 
product at each of several skill levels and thence computing skill-based 
productivity measures and mean skill level. 

GENERAL RESEARCH APPRO ACH 

Established job-evaluation schemes used by firms for deriving basic pay 
rates provided the essential data for assessments of skill-level, a quantita- 
tive skill-scale being obtained by srouping point scores for selected factors? 
the point scores for a single process fall into a distribution termed a 
skill-profile." Mean skill-levels were computed from the skill-scale point 
scores multiplied by man-hours per unit product and summed over all workers 
employed on the process. The standard deviation and range of skill-levels, 
and other relevant statistics were also computed. This enabled comparisons 
to be drawn between the skill-profiles obtained for two or more levels of 
production technology. 

ANALYSIS 

Case material was carefully selected and controlled for the comparisons 
to reveal only the differences in manpower and skill requirements dun to 
changes in process technology and not to other irrelevant factors. Two (or 
more) processes were matched in respect of the dimensions, quality specifi- 
cations and other relevant characteristics of raw materials and finished 
product. Processes were in a steady state, with a criterion that two or 
more years' productive operation had been applied to avoid transient effects. 

Random error (uncontrolled variation) is needed to test observed dif- 
ferences for statistical significance. This was obtained by replicating 
each matched-pair comparison in a different firm or organization, matching 
the raw materials and product as closely as possible, and also technologies 
of the older and newer processes operated by the two (or more) firms or 
organizations. 

When controlled data for two or more such pairs of processes have been 
obtained it becomes possible to test observed changes in mean skill-level for 
statistical significance by analysis of variance of the roan-hour data. Vari- 
ations ("main effects") due to skill-level, technological level and firm are 
removed first, and the interaction between skill-level and technological 
level is then tested for significance against the three-way interaction be- 
tween skill-level, technological level and firm. Thus, fully quantitative 
conclusions can be drawn from a small number of sample observations, provided 
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that a minimum of four matched processess are studied (i.e., two or more 
technological levels in each of two or more firms) . 

FINDINGS 

1. In each case the introduction of more highly mechanized production 
raetht>ds was associated with substantial reductions in total man-hours re- 
quired per unit product (direct labor). Productivity increases ranged from 
26.5% to 271%. 

2. A given process showed closely similar patterns, both of unit labor 
requirement and of change with technological advance, when operated in dif- 
f-^rent firms and organizations. 

3. Different processes showed statistically significant differences in 
skill requirement and different patterns of change with technological advance. 

4. The skill-profiles for the older technologies in banking and steel 
processes showed a preponderance of middle-level skills, while in aerospace 
machining processes higher levels predominated. It may plausibly be sur- 
mised that the absence of lower-level skills is associated with the fact that 
the older processes were already highly mechanized. 

5. The reduction in per unit labor requirements due to introduction of 
new processes was concentrated in the middle range of skills, except for aero- 
space machining where the higher level skills showed most reduction. 

6. Mean skill-levels were increased to a statistically significant ex- 
tent in all cases except aerospace machining (where they were reduced signifi- 
cantly). However, the changes recorded were all small both in relative and 
absolute magnitude. 

7. Absolute per unit labor requirements in the skill groups were reduced 
in all cases except demand-deposit accounting and galvanizing. Even in these 
two instances the increases were small in terms of full time job equivalents. 
Thus, the evidence does not support the widely expressed view that advanced 
production technology substantially raises the requirement for highly-skilled 
operating personnel. 

8. The overall impact of the advanced technologies studied was a small 
net increase in mean skill level due to a larger reducLion in middle-level 
than in higher-level skills. 

The above results were confirmed by assessment of educational and job- 
experience requirements which were found to rise very little with advancing 
technology. However, these changes could not be tested for statistical signifl- 

The diversity of skill -content (as distinct from skill -level ) was 
assessed by a count of Job-types, and the results indicate gV^r diversity 
in one case only, demand-deposit accounting. Job diversity showed little 
change in manufacturing industry. 
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CONCLUSION 



The present study has served to develop a quantitative methodolocy for 
studying manpower and skill in relation to technological change and for 
analytically forecasting employment by industry and skill-level. It has also 
provided pilot results from three distinct areas of modern industry which Uavo 
traditionally employed a direct labor force of significant size. Further 
studies in different industries and on different types of processes will greatly 
strengthen the confidence attaching to the conclusions on automation, skill » 
and manpower outlined in this report. 
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